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PREFACE 



^ The Program Standards Evaluation System was developed 'in response ' 

to evaluation requirements in the 1973 Rehabilitation Act. 'it has been 
designed with a focus on VR service outcomes and outputs, and on key 
procedural issue's. The system includes procedures for using standards 
da^a to monitor and evaluate program performance, and to assist program 
managers in directing their programs toward^ improved performance. During 
the last two years, the system of standards has been pretested in six 
Model Evaluation Unit states. This report summarizes the results of that 

pretest, and includes Berkeley Planning Associates' final recommendations ' 

I 

for refinement of the system. 

The refined system contains 13 standards and associated data elements. 
Our recommendations fox data collection procedures have resulted in greatly 
shortened client surveys, and have, attempted to integrate new data <£ol- 
lection needs into the routine data requirements to reduce burden of effort. - 

Berkeley Planning Associates would like to thank the MEU evaluation 
unit^ for their participation in the .pretest. We would also like to thank 
the San Diego State University RCEP IX for serving in its role in training 
and data analysis for the Procedural Standards Pretest. Finally, we would 
like to thank the University of Michigan .RR*!* and all RSA reviewers for 
their input into our pretest efforts. 

This pretest report is Volume I of our final report on the system. 
Volume II covers -the Analytic Paradigm , which describes the management 
uses of the system.' Volume III, the Guidance Materials , includes the 
revised forms, reports, and instructions for preparing them. Volume IV 
is Training Materials for users of the system. 
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I. INTRODUCTION- I i 

The 1973 Rehabilitation Act contained, among its many Other provisions, 
a requirement that evaluation standards be devised and implemented to measure 
the performance of the VR program in achieving its mandate .^^^ver" the last 
four years, Berkeley Planning Associates, under contract to the Rehabilita- 
tion Services Administration, has developed a revised system of vevaluation 
standards. Two distinct sub-system§'of performance measures were developed* 
One, the proposed Program Evaluation Standards, evaluates the federal-state 
VR programs. The other, the proposed Project Evaluation Standards, measures 
the effectiveness of individual projects, as well as aggregated program 
authorities funded by RSA discretionary funds. During the last three years, 
these systems were pretested in six model state evaluation units. In this 
pretest, BPA assumed the responsibilities which will ultimately be under the 
authority, and perhaps actual execution, of RSA, by providing training in 
the instruments and procedures^ for their administration , providing technical 
assistance to the states in conducting the pretest and analyzing their data, 
and providing the basic reporting of the states' performance^. This report 
discusses the Program Standards pretest experience, and provides Berkeley 
Planning Associates' final recommendations for refinement of the Program 
Standards system. ^ 

The, final recommended Program Standards ' consist of eight "performance" 
standards and associated data elements; and five "procedural" standards and 
associated data elements (see Table 1). The performance standards pertain 
to service outputs and outcomes (e.g.; coverage, effectiveness. Impact), 
while the procedural standards pertain to service method and process (e.g., 
case handling). If imj/lentented, the Program Standards would require a 
revised reporting system -for all state agencies. Tlie federal administration 
would, in tum^ generate information far measuring the achievement of overall 
program goals and for monitoring key processes which protect client interests 
In addition, the system design includes a mechanism for analyzing and under- 
standing the factors contributing to goal achievement, and for applying tho^t 
understanding in support of federal ^nd state program managers and policy 



Table 1 . - ^ 

VR Program Standards arid Data Eleraents: Final Uecommendations , 1981 




, PERrORMANCB STANDARDS AN'D PAtA ELEMENTS 

1 . Coverage 

VR shall serve the maximum proportion of the potent lally eligible target population, subject 
to the level of federal program funding and priorities among clients. 

(i) Clients served per 100,000 population 
(li) Percent' severely disabled served 

f 

2. Cost-Effectiveness and Benefit-Cost Return 

The VR program shall use resources m a cost-effective manner and show a positive return to 
society of investment in vocational rehabilitation of disabled clients. 

(i) E:fpendi tures per competitively ^ployed closure 

(ii) Expenditure per 26 closure w ^ 

(iii) Rati,?' of to^ltl VR benefits to tottffVRS^osts *T^enefit-cost ratio) ^ 

(iv) Total net benefit from VR service^ (Discounted net present value) 

3. Rehahi 1 1 tj«n5h Rate 

VR shall maximize the number and proportion of clients accepted for services who are 
successfully rehabilitated, subject to the meeting of other standards. ^ • * 

(i) Percent 26 closures ' ^ 

(ii) Annual change in number of 26 closures " 

4. Economic Independence r 
Rehabilitated clients shaU evidence economic independence. 

(i) Percent 26 closures with weekly earnings at/aboye federal minimum wage 

(ii) Comparison of earnings of competitively employe^ 26 closures to earnings of employees 
in state . v 

5. Gainful Activity ^ ' , ♦ . ^ 

Tliere ^hall be maximum placem^t-'6T rehabilitated clients into competitive^ employment . 
Noncompetitive closures shall represent an improvement in gaxr\/ul activity f<^r the client. 

(i) Percent 26 closures' competitively employed - ' » ' 

(li) Percent competitively employed 26 closures with hourly earnings at/above federal 

minimum wage , » 

(ill) Percent noncompetitively employed 26 closures showing improvement, in function and 

life status (implement after FAI/LSI pretest) 

6. Client .Change ^ — / ^ 
Rehabilitated clients j^all evidence vocational gains. * 
(i) Comparison of earnings before and after VR services 

(li) (In addition, changes in other statuses, and functioning ability, when such' measures 
become available) 

7. Reten.tion 



Rehal)i li tated clients shall retain the benefits of VR services. 

(i) Percent 26 closures retaining earnifigs at follow-up 

(ii) Cojr.parison of- 26 closurcb with public assistance as primary source of suppqrt at 
clo^^ure^ and at'follou-up ' » . ' -/ * 

(m) Percent noncompetitlvelv employed 26 closures retaining closure skills at,follow-up 
(implement after PAI/LSI pretest) 

Sat 1 <;f.>ct ion 

Clients shall be satisfied with the VR program, and reliabi 1 i tated clients shall appraise 
VR services as useful m achievi*?g and naintai ning, their vocational objectives. 

(i) Percent closed clients sat»isficd ^yith overall VR experience 

(ii) Percent closed clients satisfied with: isounselor, physical restoration, 
job training services, placement servicesX 

(lii) Percent 26 closures jxidjiing services received as useful in obtaining their job/ 
homemaker situation or m current performatice 



12 



' Tablp I (cont .) 



PROCRtHJRAb STANDARDS ^ 



9^ R-400 Vili^itv . ^' ^ • , . ^ 

Jaforrnation collected on clien ts by .th9 R-SDO and all data reporting systems used by 
shall J)e vali^,' reliable, accurateTa^ral^ftDlete. % 



Id. ea^igAiitv 

" . Elyjnbility decisions shall be based on accurate and sufficient diagnostic iriforlh^tion, 

and VR sMU continually review and evaluate eligibility decisions to inSure that 

clecisions are being made in accordance with laws and regulations, . ^ ♦ 

*• • • 

'/ * VR shall s'lnsure that eligibility decisions ^nd client rtioveraent through the VR orocess ^ . 
• * occur in a timely manner ajjpropriate to the needs and capabilities of the cli^Vs. 

^ % *• .* * 

12. IWRP • " « _ ^ ' 

VR*shall provide an individualized Written Rehabilitation Program for each ^j^jcable 
client and -VR and the xlient sj^aij be accoyntable to each other for compiyinaSS^.fi 'jjfis ' 
agreement. / ^"N. * , 1 

13. ^a^l^^Planning . • r - , . * 
' • CounseTot^ shall make an'cffort to set n^listic goal's for- clients. Comprehensive con- 

^ sideration ^I3ust be given to all faCV^'fty in developing api^ropriate vocational goals- such 
that there is a maximum of correspondence" bei^een goals and outcomes: competitive goals 
should have competitive outcomes and noncompetitive goals should have nonc^petitiye 



- %i — 

'ate7amnfcKU« 



outcomes . 
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makers in decisions regarding changes in program procedures and policy, 
state needs fqr technical assistance, arid program, needs -for. further invest 
igation. 



on. / • 1 ; ' ^ y 

This report discusses a wide variety of issues ^ith respec/^ to the ^ ' 
Program Standards system and the Program Standards pretest. In Chapter 11, 
we present an overview .of' the development o'f evaluation^ standards in VR, 

give the reader an understanding of, the context out. of which the revised 
standards evoiv^. . " . ' i ^ . 

ChaiJter III 'deals with the uses of standards data as a tool Jn^managing 
VR-prog:fkm ojierations. First, we 4iscOss our impressions on the current 
uses made by management personnel. ^ Following this, \^ present a comprehen- 
sive podel/which describes how standards data c^n be used -to inform decision 
making within the VR program. : 

In Chapter IV^ we begin our discussion' of the pretest results, focusing 
on- the eight performanba' standards . "ffsxe, we discuss: * V> ^ 

• * tiie rationale for including ^the Various standard and data 

elements jin the overal Program Standards system; 

• the comments received from reviewers in RSA, the MEUs, and 
* othjsa:. state agencies; and 

• the results o£ tke pretest dat^. 

Finally, we present our response to the reviewer ,cpmments, and end with our 
. recommendations^ as to' whether the standard or data element should remain in 
the system as is, remain in a modified form, or be eliminated from the sys- 
tem. 

Baled on these recommendations, we turn next, in Chapter to the 
'Performance Standards data collection forms. Here, we discuss such issues ^ 

• the training and^ implementation procedures involved in the 
particular datrf^ collecTien activity ;• < 



r the ij^ticipants' evaluation of the pretest training activ- 
ities^ ' ' > » ^ • / 

• tine investments by the participants in conducting the pre- 
test activities; and . 



findin"gs on the level ^of missing data. 
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Again, as in Chapter IV, we end each discussion with our .recomm^Itjdations 
for the data which v/iil be needed t-o address the revised ^sjtandards . 

In Chapter VI, we digress somewhat from the standards themselves, to • 
^ provide working illustrations of the analytic model developed for use, by 
progrim managers. Using Performance Standards c}ata from the pl-etest, we 
show how agencies can investigate the reasons behind^their problematic 
performance, and take steps to alleviate the^problem. J 

In Chapters' VII, VIII, and IX, we return to^the standards system it- 
self. These chapters coi>cem- the Procedural Standards.' We' treat the 
Procedural Standards separately from the Performance Standards, for several 
reasons^ For one, tl^e substantive foci of two sets of standards differs: 
whereas "the performance standards pertain to s'ervice outcomes, the procedural 
standards pertain. to service method and process (e.g., case l)andling). 
Second, ^he two systems require ^very different data collection methods. 
Finally, responsibility for much of the aotual pretests (development of 
forms, training, data, collection, and analysis) was spld;t> between two organ- 
iiations, with BPA conducting the performance standards pretest and with the 
procedural stand^ards pretest beingit conducted by oui; subcontractor, the Case . 
f Review Project within Xh^e San Diego State University RCEP. However, our . 
purpose vis-a-vis the Procedural Standards is the same as for the Performance 
Standards: to arrive at final I'ecommendations for system design. ^ In 
Chapter VII, we present an overview of the Procedural Standards. Chapter 
VIII discusses the pretest results, including time investment by MEUs, ' 
reviewer comments, our response to those comments, and the pretest data 
results. Chapter IX summarizes the various issues raised during the pre- 
test, and draws out the implications of those issue§ on a final recommended 
system design. 

Finally, Chapter X summarizes the various changes we recommend making 
to the standards, data* elements, and da<a. col lection forms in order to^ 
have the most useful and effective Program Standards system. 

Overall', the standard^ system has been received favorably, and has 
benefited from review by many RSA and Vocational Rehabilitation researchers 
and administrators. 'We feel that the system presented herein will make a 
valuable contribution to the VR field as a topi for evaluation, management, 
•*^^and program improvement. . ^ 
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' - II. DEVELQPMENT OF EVALUATION STANDARDS . ' . ^ 

^ IN THE VOCATIONAL REHABILITATION PR0GRAJ4 

' ' ' Ln' ■ ■ ■ " ' 

One of the most -frVc^^tly cited prcrblems in the development of performance 
indicators in the social services is' the difficulty of identifying outcome ' 

. indicitbrs* In the Federal -State Vocational Rehabilitation (VR) ^)rogram there 
is a long tradition in the use of a single outcon^ measure — employment of a 

"i^ disabled client 60 days after service and placement as the indicator of pro- 

^ gram success^ Thi^ measure, while easy to understand, has been criticized as 
, encouraging ^Iheaniing,*' or selecting only those clients who can most easily be : 
trained and placed. An effort to develop more responsive measures was spurred 
by 1973 legislation which called for the development of evaluation standards 
for the program. One set of standarfis was developed and tried, with states re- 
porting standards information to the Federal Rehabilitation Services Admini-s- 
tration. Based on critiques of the^ initial standards', RSA began an effort to 
develop and refin^ a new set 'fef standards, Th^s new set of standards has been 
pretested, and' may become the reporting requirements and performance meas.ures 

• for the state VR agencies. ^ In terms of validity and ccfricept, the^e standards 
have enjoyed a better reception tha© theist predecessors; yet, the ultimate^ 
test; will be in whether the standards are useful ip the ijonitorin^ and 
injprovement of perfoAnance in the^ system. 

DESC-RIPTION OF THE SYSTEM: HISTORICAL, BACKGROUND 

The Federal -State Vocational Rehabilitation (VR) Program provides resources 
to disabled /.persons who confront handicaps, and have vocational potential. The 
VR Program became law in. 1920; passage of the legislation was assisted by the 
compelling' economic argument that a self-supporting citizen was preferable, in 
terms of the national welfare, to a disabled person who was dependent up9n 
^ public support. Initially, the legislation was concerned with providing medical 
services to the physically disabled which would enable them to find jobs. In 
subsequent amendments to the legislation, the scope of eligibility and services 
was expanded to include services to the family of the handicapped and to cases 
of psychological disorder, alcoholism and drug abuse. The Rehabilitation Act 
of 1973 included a mandate to serve the severely disabled, those, with the mos,t 
handicapping conditions and in n^ed of more intensive services. 

Consistent with the historical emphasis on employment, the success^ of 
service ^to a client has been measured by wfietheT or not the client is "closed 

0.. 16 ^ ^' ^ 




rehab 34! itated*' or placfed in a work situation for at least 60 days after closure. 
Competitive einplo,yinent has been the favored placement, but "success** may also 
be clainedfor placement in sh^eltered employment or in a homentaker or unpaid 
family worker situation. These last two ^e considered ^successes 'because per- 
formance of these roles jnay fr^e other family members to enter the work force. 

The 1973 Act also contained a provisio^f-* calling for deveropmerlt and use 
of ^program perfortnance standards. The Act provided that ^ 

"The Secretary shall 'develop and publish general standards , ^ 
for evaluation of the programs and projecfeffectiveriess in 
achieving 'the objectives of this Act...'" [P.L. 93-112, 
Section 401(33(6)] ^ y ^ f 

As the program shifted its priority to serve thV iftore severely disabled, 
successful placements, particjilarly in competitive employment, became mpre 
difficult to' earn. .The progxam confronted inflationary costs at a time when 
its meas^e of productivity reflected idss effectiveness. Within the program, 
an interest developed in examining alternative ways to measure performanp^ and 
to direct the program, f . * 



THE 1974 STANDARDS ' . ' , • 

' . . ■ • ^.^ ^ . 

'Tli^^l9*75 Ac^ called for publication, of an initial set of standards by 
July 1, 1974, less than 10 month^ from the effective date of the Act (P.L. 
93-112, September^ 26 A49737 Sec. 40-1 CS) b). Ttie first stjandards, published *' 
in 1974, were prepared by RSA and reviewed by members of ^Ijie Council of State 
Vocational Rehabilitation' Admini?t^ators (CSAVR) . These standards identified* 
a number of features in^the rehabilitation process. Because of the ti'gh't time 
frame between the legislative mandate and the scheduled time foi> announcement^ 
these standards and their measures were not tested with state data; at this 
point, debate focused on the conceptual issues and on knora results from prj 
vious studies or state experience. ^ 

. Hie performance of each state agency was to be compared against other 
agencies. States would learn about their comparative performance after eath • 
state result w^s included in the standards analysis. Separate performance 
levels would be set for agencififs who served onlythe blind. 

The standards published in the Federal Register were as fallows; 



1. To insure ^hat the rehabilitation program is.^rving the 
eligible disabled population a'nd that these services are 

* -provided in an equitable manner • 

2. To insure that rehabilitated clients are placed, in gainful ^ 
employment suitable to tfieir' capabilities. 

3^ To insure that undue delays are avoided in' j/roviding clier^ts • 
with VR services. 

4. To insure that available resources ar^ utilized to achiev^ 
, maximum operational efficiency. . ' * 

s/ To i'nsure that managdable-size4 caseloads are maintained. . • 

6. To insure, that clients olosed rehabilitated retain the * • 
benefits obtained from the rehabilitation process. 

^ ^ 

7- • To ipsure that the need fbr post -employment services is 
satisfied. 

« • . , - * 

8. To insure that agencies ape**consistently- identifying reasons 
why clients are not successfully rehabilitated. "jh 4 

9. To insure ^at the client is satisfied with the vocational 
rehabilitation services as developed with the counsel^. ^ 
(Federal Register, Vol. 39, No. 128, July 2, l'^4.) ( 

For each of these standards; which focus heavily on compliance with 
the spirit .and management of the rehabilitation process, *'data elements" 
or statistical meas\ires driwn from regularly reported client and program 
data were prepared. The norm fq^ performance- on most ertments was ^et as * 
plus or minus one standard deviation from the mean performance of all state_ 
VR agencies (plus or minus depending on whether a minimum ox maximum value 
for^ tjre element was desirat)le.) Such an approach meant that for data elements 

m 

ith a normal or near -normal distribution,, about 16% of the Estates Would be 
"out of complaince." These standards were promulgated in the Federal Register 
' with performance level's based on the year's pe?"formance. 

States wore required to send data submittals to RSA. States experienced 
• a variety of hardships in preparing the Required data ^r the standards , due 
to thkilr varying data proce$sing capabilities. V/hile same states were able 
to liijndle these requirisments through automated processes), Others had to do 
hand calculations. Also, some states had their VR agenpes located within 
social services "umbrellas." For these agencies, there was often difficulty 
in obtaining their share of data processing time or •^software support to 
respond to the new reqirementjs. TItere was no direct technical assistance 
available from RSA, even though RSA called on states to make tha submittals. 
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The promulgation of the standards gave the states thi;pe implied 
^ evaluation changes: • * . , ^ , • . i 

J^i bevelopment of capabilities^ for estimating the e*ligible disabled 
peculation (size and charact^eWstics of the eligible population). 

(2) ^Development of state, capacity for evaluating the effectiveness 6f 

the state agency ^service process. ^ ' 

(3) Development o^ capabilities , for determining thp extent to whiclf 
•the Rehabilitation agencies ,were achieving"thei!r. objecti-^ew 

^This call for program Evaluation, capacity faced states with the challenge 
of developing some program evaluation strategies for meeting the annu.al audit 
demands cfea^ted by^^the new legislation (Rubin, op.cit.). To assist states 
ia^responding to these -newdemands , pSA awarded a grant to ihe JJniversity of 
Arkansas'^ Rehabilitation Research and Training Center. Th"e Arkansas Project 
contained three phases. * First, there was a ^National Program Eva/Ju^tion 
Conference in Memphis, Tehnessee^ in October of 1974. This conference brought 
evaluators' from state rehabilitation agencies, RSA regional office^^ and ufii- 
Versities 'to enchange ^deaS about program evaluation, and to develop study 
groups to examine specific methodological aspects of the RSA standards. Phase 
Il/was the study phase, when participants. examined available data aad methods 
for implementing the evaluation'^requiresl by the published standards. *There was 
a study team for. each of the^ 10 RSA regions. The-final phase was a meeting 
held in New Orleans in April of 1975. In this conference , the findings of the 
regional teams were presented to f nd discussed with "state agency program 
evaluators.' \ , ' 

These studies, focused on|^onceptual and ineasu5rement problems ip th^ 
standards, and examined alternatives to 'the de1;€rmination of such pi'ogram 
concepts as: ^ * * ' ' \ 

• ' estimating state dibbled population ; . , 
^ ' ' • ^determining "hianageable' ca'seload size"; 

• measrurin^ client satisfaction. • ^ • ' - ^ 



-SanfordlE. Rubin (ed.) Studies. in the Evaluation* of State Vocational 
Rehabilifation Agepcy Progarams, Arkansas RehabJ.litation Research and Training 
Center, University of Arkansas, November 1975. 

2 • ' ^ " . 

These findings were ^cornpiled in a series of Working papers commonly 
referred to as the "New Orleaijs Report." 
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States themselves, .J.n responding to the^ first ann9unced s'tandards, 
identified definitional issued, gaps in -existing program data, and 
questioned .the selection of the various^ standards and their measures. 
The performance on these standards was based on state norms ,^ and compara- 
bility on many items was questioned • ^ ' ^ ^ 

The states gained experience on; these standards, and over the -years*, 
the definitional issues have been clarifie.d and report formats developed 
andlrefined. This initial set of standard6 is. still in effect in 1981 as 
the basis for'state -r'eporting on^ the standards* 

IVhile the original ^et of standards met the. requirements for reporting, 
set forth in t^ie Act, kSA sought further development and refinement of the 
performance measur^ent. This first effort, while prOvidi^ a beginning in 
thf reporting process and "in the idea of evaluation or performance standards 
did not meet the co^peptual test of expressing" the goals and decision points 
of thfe program to. the satisfaction of tnost of the actors Involved. So', a 
deyelopm^tal, activity was supported; the aim was t6^ leaJn frojn experience- 
with the .first standards in order to develop a better s>jstem*r 

STANDARDS DEVELOPMENTAL. WORK; URBAN INSTITUTE . ' " 

: — 

In 1975, RSA contracted with the Ufb^ Ins^titute 1X> use 'a iftuch.more^ 
analytical approach to refining the standards.* The Institute had proposed 
the .development of a simulation mo^el of the, rehabilitation system, aYid 
the ultimate-setting of standards performance level's based on analys^is * 
using the model. < , ( 

There were a number of analytic problems to be solved in bui/lding 
such a modei* Of the many'VR processes, prqject and program activities, 

the Urb%n Institute finally cho^e to focus On eight '^dimensions" qf client 

^< • 

flow through .the 'program relating to client intake, program services, and' * 

« • ^ * ' ' ' ^ 

program outputs: ,' ^ ^ ' ^ 

• outreach ; • > ' ' 

*» • * 

• referral ;< ' . ( - ^ * 

• client mixt ^ ' * ^ - 

• service utilization; ' • , ^ ' 
, , • facilitibs utilization; 

. r • . ' . 



• timeliness ^ ^ , ^ 

J •* similar benefits (Services obtained^ from other, programs^; and 

• ' oirtcome mix. • ^ - ' - 

The Institute staff examined each of these topic areas in a series of 
"issue paf)ers.'^V. For each of the topics, the institute suggested further * 
analysis, mostly iij the^ form of multivariate regression, on specific sy^stem 
question^ as they emerged in issue papers. The Institute's recommendations 
typically called for procuedures to ensure adequate performanpe. For -exampl 
in Outreach and Referral , the ^Institute suggested two hypothetical standards 



''If for any quarter, applicants to an agency do' not contain 
sufficient pei^sons of priority characteristics'^^to assure . * 
caseloads mix objectives are reached, measured b^; a model of 
> desired proportion in caseload and rate of case closures, 
.the agency will exercise outreach efforts." (Turem, p. 43.) 

"Each state agency "will .i:onduct an. annual analysis of its 
* ' ref^ral sources to determine whether such sources arfe sending • , 

^ suitable clients* for services. An agency or organization having 
^ high rate of referrals who are not accepted, or i5 acc^ted, 
• are not rehabilitated will \e provided with specific written 
guidance on which clients have been referred improperly, why and 
what to 'do to reduce^ this rate." (TJrem, p. 44.) 

The Insj:itute, in its final reports, criticized the exj^sjing standards 
system and recommended development of sophisticated statistical^'techniqueS 
neeie^ for ^^mparison of ^tate programs leading to a comffrehensive micro- ^ 
simulation or "overall eyaluation framework." (Urban Institute, « 1976 . ) 

^ The Institute effort resulted in focusing RSA and state attentiofi on a * 
numbpr of conceptual and analytical problems, and issues inhereijt in the' 
development of standards. As k developmental study, it did not result in 
'changes in the existing standards. • \^ . " * • . 

STANDARDS REVISIO'N: BERKELEY PLANMING ASSOCIATES 

In the Fall of 1976', RSA again called for furthjer developmen|^ and refine 
ment o£ the standards through a contract with Berkeley Planning Associates 
• (BPA). This work was to drav; from the previous efforts, but called for a new 
conceptual apprfe^c^ to the development ^and refinement of the standards. 

By this lime strettes liad prepared one submittal in the RSA standard 
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\ ■ • ' ■: 

and were jSreparing to submit another • BPA began its developmental effort 
by discussing the standards with administrators', researchers*, and data 
per^orrttel ip several state rehabilitation agencies, to^determine s^tate ageiTcy 
vie*^ of the rel-evance and usefulness of the current s^ndards, to identify^^, 
standards implementation issues i^^tat-e program's, and to identify state ^ 
agency views relevant to continuing'development and design of .the* VR standard's. 
At that time, BPA found little evidence that the standards were used or con- 
sidered useful by state personnel. On the contrary, we found that^: 

• There >m^been almost no impact on basic ser\^ce delivery as a 
result of implementation of the nine standards, 

. • Rehabilitation counselors- did nt)t typically know about the standards. 
1^ • For the most part*) the standards were perceived in 'the States si^l>i y 

as a federal reporting requirement, not /as useful in the states' 
own planning'^and evaluation processes, 

• Tlie implementation of the standards had been complicated by incomplete 
* • ' instructions on methods and definitions- to usi, and by untimely,.' 

\ sfibstantive feedback by RSA to the states on their submittals,*" 

• State administrators did^ no't .stress the standards in tlieir agencies. 
Rather, the standards .remained the -domain of research or evaluation 

, ^ staff without direct, connection to management, 

• The states had no clear sense of what RSA would do with the data 
submitted' on the standards, or what direction the standards system 

• might take, ' • 

^ \n short, the standards had not as yet succeeded in their intended 
purpose: to assist RSA and the states in ra.easurin^ performance and directing 
the program towards improved performance. Some of these problems have since 
beei\ alleviated.^ For example, RSA now provides standards performance reports 
and other feedback to states. In point of fact, the short time frame for, 
developing and implementing the standards had not allowed RSA to develop , the 

\ reporting systems or technical assistance capacity necessary for smooth state 
implementation early on. State submittals, once ^ prepared, were sent to 

^Washington. However, the central office, did not have'the staff resources 
to immediately process submittals, or even to^ review the submittals and answer 
substantive questions on^ their contents. Eventually, this problem was 
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alleviated through contracts, first with JWK and later with the Rehabilitation 
Research institute at the University 'of Michigan, for statistical analysis of 
the submittals and. the preparation of state reports. Still, problems exist 
in this feedback mechanims. First, the reporting system^oes not operate in 
a manner timely enough to make the standards useful for program managers. Even 
six years after the initiation of the requirement, the lag time fbr, reports 
was such that the 1977 and 19?8 information was available on in the summer of 
1980. The 1979 information was^ available in the summer of 1981. This lack of 
prompt feedback has limited the utility of the reports for managers. 

Th^ reports themselves limit the usefulness and the distribution of the 
standards. The standards are pub]|ished in sets containing one volume for eaph 
of the 83 a^cies'plus a summary volume of national information.* Con sequent l^y, 
the reports are bulky and expensive ^to produce and ship. While these reports 
are quite thorough in their presentation of comparative an3 historical informa- 
tion for each darta element for each stat^, They do not lend themselves to wide 
distribution, or to quick scans by administrators. 

r 

The Dimensions of Program .Accomplishment : Identification of Standards 

\ 

k • 

The states' reaction to the 1974/75 standards, and the clear "indi- 
cation of lack of use , pointed t6 the ne^d for reassessing the content an^ 
purpose of a performance standards system for rehabilitation. One general, 
criticism of both the published standards and the suggestion by the Urban 
Institute was 'that measures of processes, of compliance issues, and of program 
impacts were mixed together without an underlying conceptual framework 
BPA's new design effort begin with an examin^^tion of alternative conceptual 

approaches to the development of . standards, A review of standards-setting in 

2 

other social service fields showed a variety of approaches, from a focus of 
inputs (either as structural ox "gate-keeping" eligibility standards) to 
processes (measures of "best practic^*) to outcomes "(or program impacts). 
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^ VR Program Evalua tio n Standards: A Critique of the State of the Aft * 
Berkeley Planning Associates, Berkeley. California, January 7, 1977, p. 158, 

^Ibid, pp. 42-48. 



After analysis of the strengths and w^knesses of • alternative approaches, 

an approa<?h which emphasized program outcome was recommended. "This approach was 

favored because BPA felt that to guide performance and measure achievements, 

"standards and their data elements (measures) should be expressive of, and 

2 * 

allow measures of< achievement of overall program goals J' The designers 
identified three levels of questions regarding pjogram activity, as shown in 
Table 2. ■ • ' 

BPA postulated that compliance concerns were best handle through audits 
and other review procedures, and that performance evaluation should focus on 
the second- and third-level questions measuring and relating to goal achieve- 
ment., Many compliance questions relate specifically to whether or not the 
regulat/Lons are being followed to the' letter (e.g., is there a client signature 
in the written plan? did the counselor record the date of plan completion?), 
^rather than program ^effects. Concerns at this level detract from a focus on 
outcome by concentrating on processes. BPA thus argued that standards and their 
data elements should be developed only for those dimensions which directly 
measure program outcome. Other evaluation, monitoring ajrnl research activity, 
supportive of the standards, could -be carried out to investigate areas of pro-"^ 
blematic performance. ^ J 

To identify \the di/nensiods which should fall into the performance standards 
category, BPA analyzed the rehabilitation regulations and administrative manuals 
and developed a much more detailed* model of the .client flow process (summarized 
in Figure 1). This model identified over ^70 decision points. For each point 
in the detailed model, program problems and decisions were ^^isted. These 
problems and decision lists in turn led to the development of an initial set 
of candidate areas for standards. An example^ of this analysis is shown in 
Table 3. Through use of this process, 71 candidate areas for standards were 
identified by BPA in conjunction with RSA and an advisory committee represent-"^ 
ing the'council of State Administrators of V^jg^ational Rehabilitation (CSAVR) . 

1 ^ 

Alternative Conceptual Approaches to Standards (Working Paper #2), 
Berkeley Planning Associates, Berkeley, California, Marctj 25 1977. 

2 ^ 

Recommendations for VR Program' Evaluation Standards , Berkeley 

Planning Association, Berkeley, California, September ■ 30, 1977. (Revised) 
p, 5. J 
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Table 2 



Levels of Program Evaluatibn Activity* 



Question 



Evaluation Activity 



Is the actiritv ii) compliance 
with the regulations? (Whether 
or not it-affects outcome) 

2. Does the activity contribute 
to the achievem^ent of' 
program goals? 



Does the measure directly 
reflect program effectiveness, 
coverage, or impact in* meeting 
stated objectives? 



Compliance audits, case 
reviews. / 



Program Monitoring, 
evaluation^ and' research 
on effectiveness of 
processes 

Standards for program 
performance 
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-= 5(08) ^i^sed not ' accepted 



Service interrupted while 
in 14, 16, 18, 20, or 22 



no 



30J Closed after accep- 
tance and before 
IWRP IS initiated . 



Provi-sion of Services and 
Training According to Plan 



Counseling and Guidance 
i::^l6)y Physical und Mental Restoration 
ns) Training 

t 
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0 




In Placcm^t After 

• Sixty Day^? 



rehabi Utated 



_^@) Closed without 

successful place- 
' mcnt after IWRP 
initiated 



Table 3 



fexample Candidate Area Identification for a VR Decision Point 



Decisk)n 

Point Client Provided Serviced Accbrcfing to IWRP (Written Plan) 



Problem(s) : Clients may not receive a^l the services specified in the IWRP 
in the amounts and in the time schedule sp^cifi^d, thereby 
rendering the«4eA/ice plan less effective and breaking the 
agency's agreemei^:. , * 

Decision (s) : Require agencies to inv^tigate cases where movement through 
statuses exceeds^ time norms to determine whether there is a 
problem^eeding redress. 

Candidate Agencies shall insure that clients move through the 
Area(s): rehabilitation process in a timely and coordinated manner. 



Berkeley Planning Associa'tes, Speciticat ion of Candidate Areas for 
Standards, and the Conceptual Approach Underlying the Standards Systepi 
(Working Paper 5), 1977. This example is from Appendix 1 ''Elements of ' ^ 
Detailed VR Client Flow HodeJ: Problems, "Decisions , Candidates for Standards, 
P- -16. . . - ' ' • ^ _ 
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^Vhile the process of moving from analysis of the client flow to the 
identification of candidate areas provided the opportunity^ to be ''compre- 
hensive/* the resulting plethora of possibl^measures would likely obscure 
rather than clariSy the measurement of , performance . If administrators were 
to become key users of the standards information, this set needed to be 
reduced to the essential performance pleasures for the program. Therefore, 
each candidate area was .analyzed with respect to such, criteria as conceptual 
design, measurability, management utility, and ease of implementation. 
Table 4^ists the various criteria u^ed. These analyses explored each 
measure's strengths and wea*knesses on each criterion, documented any existing 
related program research, and presentecl)recommendations for the role of the 

.candidate area in the standards! system. For ea^h of the 71 candidate areas, 
project sfaff , RSA an^ the advisory committee discussed whether these program 

^dimensions should be regarded as performance standards, procedjiral standards,^ 
or supportive evg^luation elements. ^ 

Performance standards would measure the achievement of a desired outcome 
or mission of the program (e.g., competitive employment closures). Data ele- 
ments on the standards would be reported annually for each VR agency. 

t 

Procedui^^ standards would address protection of client interest by 
ensuring key processes rather than measuring ultimate program performance. 
No standardized performance level would be set due to jLjegitimate individual 
differences among clients (e.g., the "timeliness" concerns expressed in 
Table 3) . Performance on these standards would be measured through stipulated 
procedures (such as review of a sample of cases "^annually) execu'fe.^d on a 
regular basis. ' ' 

Supportive evaluation elements are aspects of the VR process useful 
in 'the analysis^, of performance to e^tplain differences and to help identify 
program actions to enhance performance on standards. These are the indepan-^ 
dent variables in causal models of the program, wh^re performance outcome (0) 
is seen as dependent on a^numher of environmental (E) , client (CJ, and 
process (P) characteristics. 0=f (E,C,P3. Data for these elements might 
come from routine client data and/or specia^ studies: 

y The analysis of the 71 Q^ndidate areas resulted in 11 candidates foj; 
performance standards, 7 for procedural standards, and 17 areas for 
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Table 4 



Criteria foV Standards Deveropment 



Criteria for Con ceptuaT Soundness 

T ' ' 

Appropriateness 
Validity 

Uniqueness (non^redundancy) 
Completeness ^ 
Intema-l Consistency 
Conprehensiveness 
Policy Consistency' 
Flexibility 
Compliance' Adequacy 
Exp'licitness of Assumptions 



Criteria for Measures of Concept 

Methodological Utility 
Data Qpality ^ 

Criteria for Management Utility 

Efficiency 
Evaluative Utility 
. Controllability 
Cost Effective'fjjpss 



Criteria for ^se of Implem^ntatioa 

Clarity 
os^ 

Controllability 
Face Validity ^ 
Capability \ 



/ 
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supportive evaluation. The remaining 36 original candidates were dropped 
' from further consideration as having no direct relationship to program 
mission, as being a compliance concern, or as failing to meet system design % 
criteria. Next, BPA conducted further analysis, refinement and reduction to 
achieve a balance so that individual standards would complement one another • 
This was nece^sjlry because goals of the program were in tension; for example, 
serving more clients (coverage), was ^l^etter'*, subject to overall cost 
(efficiency,) - ' Obtaining high numbers of successful closures (impact) -was' 
desirable, but should be done with concern for service coverage and cost. 
The program trade-offs between coverage, efficiency, and impact are suggested 
in Figure 2. This further refinement reduced the number of performance 
standards to eight, with four addition^ procedural standards, shown in Table 
^ 5. 

/ 

■ The Measurement of Performance: Identification df Data Elements 

It is one thing to express a program goal such as "increased client ^ 
economic independence for clients"' or **use of resources in ^ cost-effective 
manner." It is another thing to specify measures of 5uch concepts. Criticisms 
from the earlier standards make it apparent that it was the measure rather 
than the concept of standard? or ^he standards the'lnselve^ which had met with 
criticism in the^New Orleans report , in the Urban Institute's work, and in 
most s.tate "critiques of the standards. The next design task for BPA was to 
recommend appropriate measures for each of the standards. The performance 
standards needed discrete measures of dif ferent^'aspects pf program perfonnance. 

Table '6 illustrates the list of data elements recommended for measuring 
achievement of ^ the goals implied by each of the perfonnance standards. For 
the procedural Standards, a-ssessment would occur not in terms of discrete 
data elements, but rather through audit-type analyses covering a wide range 
of compliance, process, and data quality issues. 

To identify the most appropriate data elements for the standards, BPA 
first reviewed the availability of data at the state and federal levels. ' . . 
The VR system 4ias an extensive client-based data system, based on stage agencies 
routinely sending data .on closed cases to RSA annually, in addition to a 
• number of reports and plans containing aggregate data. To pretest alternative 
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Figure 2 



Program Trade -Of fs 



Coverage (How M^ny Served?) 



How many to serve 
for the budget? 



Help many a little, or 
a few a great deal? 



Efficiency » 
(At what cost?) 




Cost/client vs. Impact/client 



Impact 
(With what resU: 
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Table 5- 



1978 RECgvMENPEf) PROGRAM STANDARDS 



Performance Standards 



1. VR shall serve the maxitnum' proportion of the potentially eligible target population, sub- 

• ject to the level of fedgxal progran funding and priorities. among clients. 

2. The VR program shall usfc resources in a coSt-effective manner and show a pj^itive return 
to society of investment in vocational rehabilitation of disabled clientT*^ 

3. VR shall maximize the number and proportion of clients accepted for. services who-.are 
successfully rehabilitated, subject to the meeting of other standards. 

4. Rehabilitated clients shall evidence increased ecpnomic independence. 

5. There shall be maximum placement of rehabilitated clients into competitive en5)l^Jrment. Non- 

* competitive closures shall be, in accordance with the IKRP goal and shall represent an 
improvement in gainful activ3^ty for the client. 

6. ^ Vocation a l^gains shall be attributable to VR services. 

7. Rehabilitated clients shall retain the benefits of VR servi<5es. 

8. flients shall be satisfied with the VR program, and rehabilitated clients shall appraise" 
VR services as useful in achieving and maintaining their vocational objectives. 

Procedural Standards 



9. Information collected on clients by the R-300 and all data reporting systems used by RSA 
shall be valid, reliable, accurate, and complete. 

10. Eligibility decisions shall be based on accurate and sufficient diagnostic information, 
and VR shall continually revi^v/ and evaluate eligibility decisions to ensure that decisio 
are being made in accordance with laws and regulations. ^ • » 

1. VR shall ensure that eligibility decisions and client movement through the VR process 
occur in a timel/ manner appropriate to the needs and capabilities of the clients. 

VR shall provide an Individualized ITritten Rehabilitation Program for each applicable 
client, and VR and the cldent shall be accountable to each other for complying with this 



agreejnent . 
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Table 6 , i 

SlfMMARY TABLE OP DATA ClMNTS RCCONNHNnUD BY DPA AS ^g:AS^mGS 
OF THE EIGHT PERFORMANCG STANDARDS 



Standard * 


Data 


Elements 


#1 


Coverage 


(i) 


• of National Caseload Served/^ of National VR Budget 






(ii) 


Clients setvctVlOO,000 state population (total) 




3enefit-Cost and Cost Effectiveness 


(i) . 


, Expenditure per competitively employed closure 






' f iil 


Expenditure per 26 closure 




• 


(iii) 


Benefit-cost ratio 






(iv) 


Discounted net present v&lue 

• 


HZ 


Rehabilitation Rate 




rcr(.cnL Closures 




/- 




#4 


Increased Economic Independence 


■ (i) 


* 

Percent 26 closures with weekly earnings at/above 








?^A^T7i\ TTiinimiim wrc^ • 






(ii) 


Comparison of earnings of coinpotitively employed 26 


I 

I 






flnsiirps to eaTninps O'f pmnlftv^ps in srnr^ 






(iii) 


Comparison of earnings before and after VR services 


• 


- 


(iv) 


Comparison of 26 closures with public assistance as 








irrimnTV cmiTp^ nf ^iinnOTP " AiT^ fin/4» afpoy VR ^ptvi<^a^ 




Competitive Employment 


(i) 


Percent 26 closures competitively erapJoyed 




• 


(ii) 


t_ * ' 

Percent 26 closures "with hourly earnings at/above 




f 




federal minimum wage 






r i i 1 1 








1 


non-competitive outcome and non-competitive goal 








• 

Percent non-competitivcly employed 26 closures showing 








improvement in gainful activity 




Causality 


if 


. . - T 

irfUutpcirx bun ol canixngb cnangc rrum rcrerrai cio^urc / 








of'26 closures to earnings /change of a control group I 

-■ ■ • \ 




Retention of Benefits 


• (i) 
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measurtjs, BPA used annual data tapes and other relevant sources to determine 
which of the possible measures best expressed the intent of the standard, 
which were most readily constructed from existing data systems, and which 
would be of most use to program evaluato^s and administrators. 

As an example of the measuremeJ^t problem and the direction taken for 
) resolution, consider the first performance standard, which relates to 
coverage : . 

"VR SHALL SERVE THE MAXIMUM PROPORTION OF THE POTENTIALLY 
ELIGIBLE TARGET POPULATION, SUBJECT TO THE LEVEL OF FEDERAL 
PROGRAM FUNDING AND PRIORITIES AMONG CLIENTS." 

A serious methodological problem — that of estimating those ''poten- 
. tially eligible" for service — impedes the precise measurement of perfor- 
mance on this standard. Tlios.e eligible for service, remember, are those 
with a handicap and vocational potential. No regularly collected population 
survey yields this particii^r item. Nor is it possible to derive these 
estimates through cross-tabulation or other manipulation of ^existing surveys. 

In spite of the lack of a precise measure for the target population, 
coverage was regarded as an important aspect of performance in a quality 
program by state participants in the developmental process. Therefore,>^in 
the absence of a precise estima te > two coverage ^'proxies" were idetitifed as 
measures of performance on tMs standard: 

(i) % of National Caseload Served/% of National VR btf^^^T 

(ii) Case loadpfServed/lOO, 000 state population (total). ^ 
Both of th^'se measures could be constructed from generally available * 

data, but each has a disadvantage",^ compared to the ideal discilssed above. 

Alone, each is vulnerable in that neither expresses the standards. 
^However ,_each <9oes have face validity for administrators. tVhile the first 
perhaps expresses more o£ an efficiency than a coverage concern, the second had 
already been brO(adly ij^ed by state and RSA as a measure of coverage. The 



> 
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^See Worrall, J. "Some thoughts jpn'^he nature and size of the VR target 
population,'* Bureau of Economic Research, Rutgers University, November 1974; 
Ridge, Susan Shea Estimating the Need for Rehabilitation Services , Working 
Paper '/rl821/KS009^, "Elisititute df Urban and Regional Development, University 
of California, Berkeley, 1972; Worrall, J. and Schoon, C, 'Methodologies 
for the Estimation of the VR Target Population: An Exploratory Analysis," 
in Rubin, S. ed.', opl cit.; and Nagi, S. "Estimates of Target Populations 
for State and Federal Rehabilitation Programs," Ohio State University, 
Columbus, Ohio, 1974. ' , 
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two together were seen as expressing coverage effort while providing some 
acknowledgement of the funding resources available to the agency. Moreover^ 
if usaVle estimates of the target population were to become available' later, 
BPA pointed *out that the data elements or measures for the standard could 
be refined or respecified. So long as the program mission and values re- 
mained the same, the standards would remain. However, changes in program 
knowledge or in data a^vailability or experience with use of the standards, 
might result in changes in the data elements or even additions to the 

standards themselves as measurement problems were^solved. 

*■ » 

Analysis and debate followejd in this fashion for the data elements of 
each of the other seven standards. Balance between elements, both within and 
across standards, remained an important- design consideration. 

Standards Revision; Summary 
* * 

As a starting point fot BPA's refinement of the standards, it ^was clear 
to us that there must be an understanding of the role standards Should play, 
in order that the pitfalls of the existing standards might be avoided. Based 
on considerable interaction among BPA, the Rehabilitation Services Adrainistra^on 
(RSA), and representatives of several state agencies, the following ground^rules 
\ were adopted: , 

• Standards and their data elements should be expressive of, and 
""allow measures of achievement of overall program goals. In 

essence, a standard is an acknowledged program objective — a 
^ criterion or norm against which performance is assessed. ,It 
ej^presses an expected level pf -performance whose achievement 
is. "desired,*' "good," or ''appropriate*" 

^ • ^It is important that program evaluation focus not only' on the 
impact, or degree of effectiveness of the program for indivi- 
*duals,.but also on coverlge, or the number of clients served* 
' A system of evaluation standards for program performance,^ 
thus, should address effectiveness and efficiency and relate 
^ to coverage and impact issues, 

• In addition to their function in an evaluation system, the 
role of 'standards has another dimension that of representing 
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the goals of a program, the direction in which it desires to 
move, and the conditions which represent success for the pro- 
gram. Thus, standards, incorporated with an implicit or 
explicit reward or sanction system may 'become behavioral 

incentives: .program administrators, presented with a set of 
♦ ♦ 

standards against which performance will be me'asured, may' 
begin to move programs in the directions implied by the set. 

• If there are too many standards, covering too many elements 
of a program, there will be no clarity for program direction 

c 

and attention may become too diffused, with administrators 
required to attempt to meet norms for elements which may be 
less critical than others, and which, in fact, may not make 
a positive contribution to the ultimate goal of the process, j 
i.e. gainful employment of clients. 

The refinement process for the program standards entailed developing 
standards and dat^ elements by which they, could ]^ measured, specifying 
an approach to setting performance levels and adjusting the data for across- 
state comparability, and proposing implementation guidelines to RSA and the 
states. Berkeley Planning Associates issued reports^ addressing these de- 



. sign steps as follows: 



Previous work on VR program evaluation standards : Implications 
for further refinement of standards of the work done by each 
region reported in the "New Orleans Report," and of the work 
done by the Urban Institute, JWK Corporation, and others in 
analysing the current standards and proposing improvements. 
CReport of January !|.977) 

Reactions and recommendations of selec-^ed state a^gencies : 
'Appraisal of what has been the states* experience with the ' 
standards, in reporting on them, in their effect on the state 
program and changes states would like to see made* (Report 
of January 1977) 



Readers interested in detail in this material should consult the 
primary documents submitted to RSA and available from the Evaluation. Div- 
sion. , " 
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' * — 

Review and critique of the current standards ; Analysis of how 
well current standards measure up conceptually and technically 
in terms pf RSA concerns, perspectives of state agencies, needs 
of Congress and DHEW, in the view of outside researfch. and eval- 
uation grouRS.' (Report of January 19*77) 

Criteria for developing standards : Discussion of appropriate 
conceptual^ measurement, management and implementation criteria 
which must be met for the constitution of a "good'* standard 
and set of standards, (Working Paper 1, February 1977) 

Conceptual approaches to standards development : Presentation 
of alternative approaches which can be taken in developing a 
set of standards for the VR program. (Working Paper 2, March 
1977) 

' Candidate areas for the selection of standards : Analysis, within 
the VR process, of what are -the discrete inputs, processes and 
outcomes which should be consider.ed as candidates for standards* 
Evaluated in terms of the Appropriate conceptual framework and 
criteria for developing standards," candidate stinkards were' 
selected which met the test as being appropriaite/.icandidates for 
proposed standards. (Revised Working Hv^fy^'i Kv^ust 1977) 

Recoinmendations for VR program evaluation' standards : Presenta- 
tion of the recommendations for the end of the first year, * 
(RSA-IM-78-30, February 1978) 

Data items associated with standards :-. Analysis of the avail- 
ability and validity of alternative data elements proposed as 
possible measures of the performance standards, as well as pre- 
test results and preliminary recommendations of data elements 
for further consideration. (Working Paper 4 and Supplementary 
Memorandum, March 'and April 1978) ' 

Use of composite measures for assessing'agency performance on 
the standards : Discussion^ of the strengths and weaknesses of 
developing ^ composite measure to summarize agency performance. 
(Wording Paper 4, March 1978) 
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Issues, approaches and recommendations for the development of 
procedural standards ; Discussion of the background .and devel- 
opment of the procedural standards, and the adoption of a uni- 
form case review mechanism. (Revised Working Paper 5, May 1978) 

Data collection, data validation and processing, and analysis : . 
Presentation of the recommended data collection sources and 
specific data items for each data element for both performance 
and procedural standards, along with data validation and- pro- 
cessing procedures. Preliminary review and recommendations 
presented with respect to comparability adjustments, perform- 
ance level expectations, and analysis. (Revised Workijng Paper 
5, May'1978) I 

A guide for implementation : Following data colirection revisions 
and refil^ements to the performance level and comparability ad- 
justment approaches, the implementation issues were discussed. 
These included sampling, allocation of responsiMlities, and 
the proposed scheduling of activities and reports. ' (Working 
Paper 6, July 1978) 



• Proposed demonstration projects : Design of 11 demonstration 
projects to assist in the further development of the VR eval- 
uation standards, (Working Paper 7, September 1978) 

• the VR Program Evaluation Standards-: Final Report : Presents 
^ the recommend^ standards and data elements and their coverage 

of the major decision points within the VR process. Presents 
data collection needs for each standard and data collection 
instrument design. Specifies data-validation processing pro- 
cedures, and data analysis. Presents recommendations for 
imp.lementation. (November 1978) , 

BPA recommendations on these issues were finalised with the assistance 
of RSA staff and the CSAVR representatives who had participated throughout 
the development of fhe standards and .data elements. 
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TTiese standards ancl data elements were sent to state agencies and 
other rehabilitation researchers and evaluators by RSA in 1978 with a call 
for comments and review. *BPA's development^ of the standards had resulted 
in a new design. The next step would be a test of reporting systems and 
use. 

PRETESTING THE REVISED STANDARDS: THE PROCESS ^ ^ 

* RSA had learned a lesson in the first round of standards development. 
The imposition of required standards and their data elements, without a trial 
period, had resulted in a number of very valid criticisms. Rather than 
require a wholesale change-over to the new system, RSA sponsored a pretest 
of the BPA-developed standards sj^tem as one part of a demonstration project 
to stimulate program evaluation (PE) activity in state rehabilitation 
agencies. Six Model Evaluation Units (MEUs) were created in selected state.^ 
agencies (the Michigan general agency, the Pennsylvania, Oregon, Delaware, 
and Virginia combined agencies, and the Mississippi blind agency) to demon- ^ 
strati what enhanced evaluation capacity could c^p in state management and to 
pretest data collection instruments and the proposed standards/ These agencies 
for the past two years haj^a tested not merely the refined program standards, 
but also the facility standards and evaluation systems developed by RSA and 
Walker Associates, the BPA-developed project standards, and a wide variety of 
other evaluation technologies and innovative approaches* In addition, several 
other organizations played a role in the pretest: 

San Diego State University RCEP had developed the Case Review system to 
be used for data requirements of procedural standards which were a part of 
the standards system. SDSU became a -subcontractor in the BPA contract. 
SDSU's own* training and assessment efforts involving the case revi'ew instru- 
ment began prior to the pretest, however. Consequently, the MEU training 
in the instrument, and the coordination of the case review activity with 
the BPA pretest schedule was not possible. ' * 

W^st Virginia Research and Training Center (IVVRT) became the coordinat- 
ing contractor far the MEUs, the standards, and other ^involved contractors! 
The WVRT effort began, at the same time as the BPA contract. Among the. 
responsibilities /West Virginia had in this overall effort was the convening 

of Advisory Coi^ittee meetings for the MEUs, the standards and other con- 
tractors. • ^ . 

39 



•31 



Texas Institute for Rehabilitation Research QTIRR) was included in 
BPA^>^ contract as a subcontractor for the development of estimates for 
benefit-cost models rehabilitation ♦ 

. The pretest itself began witli the refinement of the forms and supporting 
materials needed for the clearance of the pretest by the Office of Management 
and Budget (OMB) and the preparation of training materials • A training of 
Model Evarluation Staff (MEUs) was conducted in Houston, Texas. 

Following the training, MEUs participated in j>he pretest; the details 
of data collection appear in later sections of this report. During data col- 
lection, BPA provided assistance.' The collection of state data for the pre- 
test was completed in 1980, with the six participating states each p^i^oviding 
client data needed for the data elements and for th§ procedural standards. 
Perfoormance for eacli^of the six states, based on the new elements ,' was 
calculated as a part of the pretest. Data reliability and validity, as well 
as the MEU's analysis of the usefulii^ss of the data, was also reported. 
MEUs commented on their experience during the standards pretest including tHe . 
time taken to install the system in their agencies. ' | 

For tlie most part, the -standards r^eived favorable comments on their 
technical merits. There were many suggestions from the participating agencies 
regarding the refinement or more spep:ific definition of some elements of the, 
data elements. Berkel ey Planni ng. Associates has used the e^erience of the 
NEUs and other reviewers to further clarify the data elements, and to recommend 
a final set of standards for implementation. ' 

FINAL RECOMMENDATIONS FOR STANDARDS AND DATA ELEMENTS 

The new recommended goajs and data elements for measuriTlg^^HM monitoring 
their achievement ^re shown in Table 7.' This tabl^ co^^ares the, currently 
proposed standards and data elements following the pretesting and farther 
refinements with the BPA proposals of 1978 which were the focus of the VJ^" 
test and wit^ the dtjcisting standards pronjulgated and in use since the inid- - 
1970s. ' A close examination of tljje table will reveal that the changes since 
the pretest are relatively small and technical, as compared to the differ- 
ences between the BPA standards and the existing- standards . Reviewed indivi- 
dually, the standards are as fol^lows:- 
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Performance Standards 



, Table 7 
Evolution of VR Program Standards 



1981 Revised BPA Standards 



1976-1978 BPA Standards 



1975 General Standards for Evaluation 



2. 



VR shall scr\c the m*iximum propor- 
tioi\ of the potentially eligible 
target population, subject to the 
level of federal program funding 
and priorities among clients 

(i) ^ Clients served per 100,000 

population 

(ii) Percent severely disabled 



The VR program shall use resources 
in a cost-effective manner and show 
a positive returp to sooiety of 
investment in vocational rehabili- 
tation of disabled cliei\ts 

(i) Expenditures per a^peti- 
tively "employed closuTC ^ 

(ii) Expenditure per 26 closure 

(iii) Benefit-cost ratio 

(iv) Discounted net present 
value 



VR shall maximize the number and , 
proportion of clients accepted for 
services who are successfully re- 
habilitated, subject to the 

meeting of other standards 
«* 

(i) Percent 26 closures 
•(ii) Annual change ^in ntynber of 
26 closures 



1. VR shall serve the maxiraura propor- 
tion of the potentially eligible 
target population, subject to the 
16verof federal program funding 
and priorities aioong clients 

<\ 

(i) Comparison of caseload served 
to expenditures 

(ii) Clients «erved per 100,000 
population 



. T)ie VR program sh^ll use resources 
in a cost-effective manner and show 

positive return ^o society of 
investment in vocational rehabili- 
tation of disabled /Clients 

(i) , expenditure per competi- 

tively employed closure 

(ii) Expenditure per 26 closure 

(iii) Benefit-cost ratio 

(iv) Discounted net present > , 
value 



VR shall maximiz^thc number and 
proportion of clSlhts accepted for 
services v/ho are successfully re- 
habilitated, subject to the > 
meeting of other standards 

(i) Percent 26 closures 



To insure that the rehabilitation pr^rara is serving the 
eligible disabled populatToh and thar these services are 
provided in an equitable manner ^ 

% ' r 

(i) Estimate of the total population eliglible for VR if^ervice^s 

(ii) Number of accet)ted' cases served (statuses 10-30) fbi^ 
the year 

(iii) Percent of annual increase or decrOaSe in number of 
accepted casesi*served (statuses ICT-SO)* 

(iv) Number of cases closed irehabilitated*" during the year 
(status 26) ' t 

(v) Accepted cases (statuses lt)-24) as a percentage df the 
total of case^ closed not accepted (status 08) pl^s 
those ca^es accepted (statuses 1(3-24) f 'A 

' * i - ^ 

To insure that available resources are utilized^ to achieve 
maximum operational efficiency ' ^ 

(i) Average case service cost per accepted case closure 
(statuse 26, 28 and 30) where case service cpst to 
the state vocational rehabilitation agency was 
involved ,^ , ^ ^ 

(ii) Percent of clients receiving rehabilitation services 
at no cost to the vocational rehabilitation agency 

(iii) Percent distribution of totaT vo^cationjtl^ rehabili- 
* tation dollars spent for agency 'operations each 

fiscal year as reported ort tHe feSA-2 expenditure 
rejiort ^ . 

(iv) Average case service cost per^type of Vocational 
rehabilitation case service inyblving ,cost to the 
state agency, and percentage, of individuals re- 
ceiving specific vocational rehabilitation service 

To insure that Rehabilitated clients are placed in gainful^ 
employment suitable to their capabilities 

Ci) 
(ii) 

(iii) 
(iv) 
(v) 



Percent of those placed in coiipetitiye ertiployment- 
(wage and Salary earners and self-employment) 
Percent of those places in riaj}competdtive onqploy- 
mcnt (sheltered workshop and' others)- 
Percent of those place<J'a5 Ji^nemakers * ^, 
Percent of those places as-njApAid family worker ' 
Percent of those placed irwt^usiness enterprise program 



* This data element does not appear in the 1979 analysis* / 



Table 7 (cor.tinueo) 



iVrtornumcc Standards (continued) 



I9S^ Revised BPA <;tjndird^ 



1976-1978 BPA Standards 



1975 General Standards for evaluation 



5. 



6. 



Rchjbi 1 1 tared •cUenth shall evi- 
dence cconoHiic independence 

(i) Percent 26 closures with 
weekly earnings at/above 
federal mini mum wage 

(ii) Comparison of earnings of 
competitively employed 26 
closures to earnings of 
emplpyees in stated 



There shall be maximum placement 
of rehabilitated clients into 
cojrpctitive efiiployment. Noncom- 
petitive closures shal ^• represent 
an ir^roveiTfcnt in gainful activity 
for the client 

(i) Percent 26 closures com- 
petitively employed 

(lij Percf'nt cor^pc ti ti vely em- 
ployed 26 closures with 
hourly earnin^js at/above 
federal mmimuni wage 

(iii). Percent noncojr.pet i ti ydy 
em)iloyed 26 closures 
bhowjng improvement in 
function and life status 



Kehabi I i tatcd clients shall evi- 
dence vocational gains (client 
change) » 

(0 Comparison of earnings 

before and after VU services 

(ii) ' (In nddition, clianges in^ 
other statuses, and func- 
tional ability, uhen such 
mcn^urc*; hecomo available) 



-4, liohabilitated clients shall e\i- 
^ dence increased economic indepen- 
dence 

(i) Percent 26 closures with 
weekly earnings at/above 
federal rainimuli wage 

(ii) Comparison of earnings 6f 
competitively employed 26 
closures to earnings of 
employees in state 

(iii) Comparison of earnings be- 
forehand after VR services 

(iv) Comp.arison of 26 closures 
with publi'c assistance as 
primary source of support 
before and after VR services 

S. There shall be maximum placement 
of rehabilitated^clients into 
competitive emplbyment. Noncom- 
petitive closures shall be in 
acc^oi'dance with the IWRP goal and 
shall represent an in5)rovement in 
gainful activity for the client^ 



(i) 
(ii) 



(iii) 



(iv) 



Percent 26 closures com- 
petitively employed 
Percent 26 closures with 
hourly earnings at/above 
federal minimum wage 
Percent noncompetitively 
employed 26 closures Showing 
improvement in gainful 
activity 

Percent 26 closures^ with com- 
petitive outcomes or with 
noncompetitive outcome and 
noncompetitive goal 



2, To insure that rehabilitated clients are placed in ^gainful 
enqiloyincnt suitable to their capacities 

.(vii) Average weekly earnings in the week before referral 
of all rehabilitated clients, 'including clients with 
zero earnings 

(viii) Average weekly earnings at closure of all rehabilitated 
clients, including clients . with zero eamipg? 



2. To insure that rehabilitated clients are placed in gainful 
en5)loyment suitable to their capabilities ^ 



(i) Percent of those placed in competitive en5)loyment 
(wage and salary earners and self-employment) 
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Table 7 (continobd) 



Performance Standards (continued) 



1981 Revised BPA Standards 



Rehabilitated clients shall retain 
the benefits of VR services 

(i) Percent 26 closures retain- 
* -ing earnings at follow-up 

(ii) Comparison of 26 closures 
with public assistance as 

^ primary source of support 

at closure and at follow- 
-up 

(iii) Percent noncompeti;tively 
employed 26 closures re- 
taining closure skills at 
follow-up 

8. Clients shall be satisfied with 
the VR program, and rehabilitated 
clients shall appraise VR services 
as useful in achieving and main* 
taining their vocational objec- 
tives ^ 

(i) Percent closed clients 
satisfied with overall VR 
experience 

(ii) Percent closed clients satis 
fied with: information pro- 
vided, counselor promptness, 
physical Restoration, job 
training services, place- 

. ment ser<vi<:es 
fiii) Percent 26 closures judging 
services received as- useful 
in obtaining their job/home- 
maker situ^kion or in cur- . 
rent performance 



1976-1&7S BPA Standards 



Vocational gains shall be attri- 
butable to VR services (causality) 

(i) Comparison of earnings 
change from referral to, 
closure of 26 closures 
to earnings change of a 
control group 



Rehabilitated clients shall tetain 
the benefits *of VR services 

(i) Percent 26 closures retain- 
ing earnings at follow-up 

(ii) Comparison of 26 closures 
with public assistance as 
primary source of support • 
at closure' and at follow- 
up 

(iii) Perjx^nt^^fnoncorapetitively 
enlployed 26 closure? re- ■ 
taining closure skills at 
follow-up 

Clients shall be satisfied with 
the VR program, and rehabilitated 
clients shall appraise VR services 
as useful in athieving and main- 
taining their vocational objec- 
tives 



(i) Percent closed clients 
satisfied with overall VR 
experience 

(ii) Perjcent closed clients 
^ satisifed with specific 

aspects of VR 

(iii) Percent 26 closures judging 
services received to have 
been useful in obtaining 
their job/homemaker s&itua- 
tion , 

(iv) Percent 26 closures judging 
services to-be useful in 
current pcrfonnancc on the 
job/homemaker situation 



1975 General Standards fof- Evaluation 



6. 



To insure that clients closed rehabilitated retain the benefits 
obtained from the rehabiWtation process 

(i) 



(ii) 

'v) 



^^Percent o^ rehabilitated clients stilTemploxed at time 
of follow-up, specifying one year, two years, or three 
years after closur^ >« 

Percent with earnings at fOllow-up, mean earnings at 
follow-up 

Percent increase or decrease of earning^ at follow-up 
Percent of rehabilitated clients (status 26) unemployed 
at follow-up for; • less than one month, one to three 
isonths, four to six months, seven to 12 months, more 
than 12 months^ * 



To insure that the client is satisifed with the vocational 
rehabilitation services*«s developed with the counselor 

(i) Percent of clients rehabilitated throughout the fiscal 
year (status 26) and* not rehabilitated (statuses 28 plus 
30) throughout the fiscal year who express satisfaction 
with tlie following, sp<^Ci£ying one- year, ^wo years, or 
three years: 

(a) counseloit^ willingness to listen to client's ideas 
and suggestions in developing the IWRP 

(b) » adequacy of informatfbn provided by^ counselor to 
^ clients for understanding!^ their disability, 

(c) promptness in thci deliveyfy of services 

(d) kind'of training received ' ^ *• 

(e) benefits of training reC<5ived 

(f) assistance in seeking job and final employment 

(g) results of physical restoration services 

(iv) Percentage of clients contacted during the follow-up 
period who stajced they would recommend vocational 
rehabilitation to a disabled friend . X 
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Table 7 (continued) 
Procedural Standards* 



4 



1981 Revised BPA Standards 



1976-1978 8PA Standards 



1975 General Standards for Evalu^ion 




10, 



IK 



12. 



Information collected on clients 
by the R-300 and all data re- 
porting systems used by "RSA shall 
be valid, reliable, accurate, and 
cociplete t 

Eligibility decisions shall be 
based on accurate and sufficient 
diagnostic information^ and VR 
shalf continually rci^ew and 
evaluate eligibility decisions to 
insure that decisions are being 
made in accordance with laws 
and regulations 

VR shall insure that eligibility 
decisions and client movement 
through the VR process occur 
in a timely manner appropriate 
to the needs and capabilities . 
of the clients 



VR shall provide an individ- 
ualized Written Rehabilitation . 
P^gram for each applical?le client 
aTO VR and the client shafl be 
accountable to each other for com- 
plying with this agreement 



9, 



11, 



12» 



Information collected on cricnFs""" 
by the R-300 and all data re- 
porting systems used by RSA, shall 
be valid, reliable, accurage, and 
complete 

Eligibility decisions shall be 
based on accurate and sufficient 
diagnostic information, and VR 
shall continually review and 
evaluate eligibility decisions to 
insure that decisions are'being 
inade;;in accordance with laws 
and regulations 

VR shall insure that eligibility 
decisions and client movement^ 
through tfiVVR process occur 
in»a timely^manner appropriate 
to the needs^ and capabilities 
of the clients 



VR shall provide an individ- 
ualized Written Rehabilitation 
Program for each applicable client 
and VR and the client shall be 
accountable^to each other for com- 
plyin^i^ wi^h. th'ir agreement 



3. To i|isur0" that undue delays are avoided in providing clients 
with VR services . , 

(i) Average time fix)ra combined referral-applicant statuses 
(statuses 00-02) to closed not accepted (status 08) 

(ii) " Average time in extended evaluation (status 06) for 

cases closed ^not accepted (status 08) 
(iii^ Average time "from combined referral -applicant statuses 
(statuses 00-02) to accepted statuses for cases closed 
rehabilitated ^(status 26) and closed not rehabili tilted 
(statuses 28-30) during the fiscal year 

(iv) Average time in extended evaluation (status 06)" for 
cases closed rehabilitated (^tatus 26) and closed not 
rehabilitated (statuses 28 and 30) during the fiscal 
year 

(v) Average time from accepted case statuses (statuses 10- 
24) to closed rehabilitated (status 26) 

(vi) Average time in accepted case statuses (statuses 10- • 
24) to closed not rehabilitated after rehabilitation 
program was initiated (status. 28) 

(vii) Average time in accepted case statuses (statuses 10- 
^ 24) to closed not rehabilitated before the Rehabili- 
tation program, was initiated (status 30) 



*l ruccdural Staridards arc not measured with data elements, but through case review and use of designed instruments. For more informationj please 
see BPA*s Report on the Pretest of the Revised Vocational Rehabilitation Program Standards , Volume 2, Drafts 16 Jul)^ 198K 
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Tabic 7 (continued) 
^crfornuince Standards J^ntinucd) 
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19^1 Revised BPA Standards 



1976-1978 BPA Standards 



1975 General Standards for Evaluation 



5. To insure that managcablc-sized caseloads are .maintained 

(i) Number of caseload carrying counselor roan years 

(ii) * Number of authorized and funded full-time caseload 

carrying counselor positions 

(iii) Number^ and percent of rehabilitation counsel^!' turnover, 
i,,e,, hiring rate and separation rate 

(iv) , Average size of caseloads as. of September 30 per number 

of authorized ^^nd furfded c^eload carrying counselor 
positions , 

(v) * Describe the^ process, if any, employed by the state 

- for each of the fallowing functions: caseload manage- 
ment; casjeload monitoring; caseload ^review 

7, To insure that the need for post-employment services is satis- 
fied _ ^ 

(i) Percent of rehabilitated clients in the previous fiscal 
year (status 26) receiyi^ig post* employment (post- 
closUre services during the 12 months following closure 

(ii) Percent receiving the following types of post-eroployment 
' services, of the total receiving post-employment services 

(a) diagnostic and evaluation 

restoration (physical and mental) 
training 

guidance and coun'seling onl)f 
maintenance 
transportatipn 
other 



1 



(d) 
(c). 
(f) 
(g) 



To insure that agc^ncies are consistcnly identifying reasons why 
clients arc not successfully rehabilitated 



(i) 



30 closures by the 



Percent of status 08., 28, and 
following reasons: 

(a) ,unablc to locate or unable to contract or moved 

(b) handicap too severe or unfavorable medical 
prognosis 

(c) refused services or further services 

(d) death 

(e) client institutionalized 

(f) transferred to another agency 

(g) failure to cooperate 

(h) no disabling condition (08 closure only) 

(i) no <^ocational handicap (08 closure 9nly) 
(j) othe;' 



f 



(ii) 



as a percentage of the tdtal accep' 
(statuses 26 plus 28 plus 30) 



m 



caso^i^closed 



5Q 



Tabic 7 (continued) 



l*rQCC't!ural Si nruiards (continued) 



IMHI Ueviscl \\\',\ Sl.iJ.d.irds* 


1976-1978 BPA Standards 


1975 General Standards for [-.valuation 


15^^ 1 vtuMvv lot > sli.tll M.iK- *ti\ cliort 

^.t^ -hc't clients. 
(!<•: prehciii>ivc- coiis idi i at lo/i n.itbt 
be given to all I'actors in devcl-. 
^opinjj ap|>ro|»riate voCiitional goals 
SMch that there is a riaxinum of 
correspcndonce between goals and 
outcomes: coiri|)cti tive goals should 
have competitive outcomes and non- 
, coK.'j)eti t ive goals should have non-^ 
competitive outrnmes 


> 

■ ft 

• 


2. TO insure that re*habiMtatcd .cl ients are placed in gainful 
en»ployii;ctit suitable to their capabilities 

(vi) ^hose who received training related to the job famUy 
in which they were placed (as identified by the first 
digit of the Dictionary of Occupational Titles code) 
as a percentage of the total number who received 
training 
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• The fir^* standard addres^fes coverage, or the extent to which the 
vocational rehabilitation program is serving the eligible target populatibn. 
The need to ensure accessibility of services to all the eligible disabled 

is bf paramount importance to RSX and the states, ^ The first data element 

.clients served per 100,000 population provides a proxy measure of 
coverage of eligible population. ^ The second measure percent of clients 
served who are sever,ely disabjed measures achievement of the priorir 
legislated for the severely disabled by Congress. 

• ThQ^second standard addresses directly the cost-effectiveness of 
the state program's overall use of resources, "and the benefit-rcost returns 
from investrai^^t^in vocational rehabilitation services. The first two dat^ 
elements measure the Cost of achieving desirable outcomes first, expend- 
itures per competitively employed closures, and second the expenditure .per 
26 closure. The focus on competitiVely employed closures recognizes the 
policy decision in RSA that such a closure is the highest priority in the 
program for clients. The second two data elements focus on the-.tw6 ac^cepted 
measures of bertbfit-cost returns — the benefit-cost ratio and discounted, 
present value. The benefit-cost model* developed at Berkeley and used over 

,^he years by RSA and 'many state agencies for^repprts to Congress and* state 
legislatures, and which was favorably reviewed by^>^ny independent specia- . 
lists, is the model used to generate these summary data elements* The model 
is to be expanded by incorporation of subsystems being developed by the 
Texas Institute for Rehabilitation Research (TIRPf) for taking the less 
monetary benefits o^ increased functional capacity and other aspects of 
independent living into account. 

• The th;ird standard monitors the quality of service outcomes being 
atkieved by the program and uses the traditional data.elemen^of per cent of 
closures which are successful (the 26 closure) and ^ annual change in the 
number of 26 closures. These data elements have a long history of use and 
acceptance in state programs as measures of how many clients VR is success- 
fully serving. ^ - ^ 

• The fourth standard focuses on whether rehabilitated clients evi- 
dence increased economic independence , recognizing that VR's most basi6 
purpose is to assist disabled persons in finding gainful employment that 
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will permit their economic self-sufficiency. Two data elements compare the " 
wages achieved by rehabilitants to national standards (the minimum wage) 
and to state norms (earnings of emp'loyees in the state). These again are ^ 
measures of the quality of service outcomes. /' * 

• The fifth standard focuses on competitive and non -^competitive 
employment ouftcomes in order^to assess the quality of closures obtained by 
VR agencies. The first two data elements measure the perc.ent of 26 closu 
who achieve competitive employment, and among th^se the percent employed a.t 
or above the national standards of the minimfum wage. The last data element 
recognizes that competitive employment may not be the appropriate placement 
for*^ll clients, but that it still is important that rehabilitation services 

'achieve improvements in gainful activity for those clients for whom employ- 
ment is not the goal. For iton -competitive closures, then, a data element - 
measures the percent showing improvements, in function .and life status. The 
instrumentation for determining such improvements is being developed by 
others for inclusion in the MIS, and will be pretested in subsequent years 
by RSA. . , ^ 

• The sixth standard is directed at measuring client change before and 
after service. The 1978 standards recommended probing causal relationships 
between services and outcomes and judging how much of the gain exhibited by 
<:lients is really attributable to the services they receive. This was becaqse* 
both clients of VR and non-clients may show, over the same time period, 
increased eai^nings., increased levels of skills^ and other vocational gains. 
After pretesting a-range of measijres, including the il^se of coinparisoh groups 
of unsuccessful closures, data elements' are recommended which simply measure 
^'before-after'* changes in earnings and (when MIS data becomes available) m 
functional capability* These measures are highly limited for input ing caus- 
ality to VR service impact but they provide* some control for the client's * 
capability pri^r to s^ervices.-/ BPA has recommended that the mandate fbr 
establishing causality be fulfilled through p^i.ddic controlled research 
studies of clients, on a- national level as part of supportive evaluation uhder- 
taken by RSA. The complexity of such research makes it infeasible for 
completion by state programs as part of their routine, ongoing evaluation 
activitv. Thus the focus of the standard becomes one of measuring client 
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« 

change, rather than Establishing that the clients' change is due to VR 
services. ♦ ' y » 

• The seventh standard again monitors quality of service outcome and 
overall program effectiveness, and focuses on the retention Jof client ' benefits 
from VR services ove;x:.time. The data . elements draw on follow-up ^ata after 
case closure to monitor retention of earnings by individual 26 closures, the 
percent of 26 closures who remain non-dependent on public assistance as their 
primary source of, Support, and the percent of non-cpinpetitively employed 26 
closures who retain* their enhanced independent living and functional skills* 

• The eighth and last performance standard monitors the 'consumer 's 
appraisal of services client satisfaction with VR services. Two data ♦ 
elements include measures of client satisfaction with overall services and, 
various aspects of services (e.g-. , counselor promptness, the quality of 
placement services). The third data element moves beyond satisfaction to 
monitor the client's judgement that services received were useful in their 
obtaining their job or 'homemakiijg^ situation • i 

Abandoned in ti||0>.proposed standards^^ revision are those elements' in 
the existing standards which focused qn post-employment services, manageable- 
sired caseloads, the reasons for unsuccessful rehabilitg.tion, and the length 
in tii^of^th^ process. The^^^proposed new^,erformance standards* 

monitor outcomevknd cost-eff ectl^enesTj not service process. . * 

In addition to the performance s1;andards, the revised proposed standard^ 
include five procedural stanci^rds that do fc^us attgi ^ion an qp itical pro- 
cess areas and on data validity. Assessment chr^perfo^Tnance on these standards 
is to occur us'ing instrumentation and procedures (modif icatipns of the Case 
Review Schedule developed by the San Diego Statd RCEP" IX) developed ^r 
ga^thering uniform data from stat^ agencies. The procedural standards focus 
on the validity and compjleteness of R-300 data, the -need for eligibility 
decisions to be based on adequate diagnostic data and to conform to federal 
laws and regulation, the desirability thalt el/gibilit'y decisions and movement 
through the VR process be completed in a timely manner appropriate .to the needs 
of clients, compliance with the requirement tor the Individualized Written 

Rehabilitation Program, and the need for realistic goal -setting for clients 

* < • 

and adherence to the policy pf seeking competitive employment outcomes when 

feasible. , ' ^^^^ , , * 
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Having completed the development of the VR Program Standards, the 
next task is to provide a system for collecting and reporting standards 
data so that it can be used by program managers within KSA and within state 
agencies. Fortunately^ development of such a system will not have to 
begin from scratch, thanks to an RSA sys^tem development effort currently 
underway. Specifically, RSA has contracted for the development of a 
Management Information System (MIS), which will include the standards infor- 
mation ^and which will provide program managers with the capability to measure 
and analyze performance on the staijdards. We next discuss the role the 
standards will play in the MIS. 



cand; 



THE REVISED STANDARDS AS THE CORE OF A NATIONAL MIS 

In 1978, RSA entered into. contract with Abt Associates to- design the 
MIS. This system was to provide RSA with the capability to produce in 
a timely and useful way the information needed for RSA's monitoring, 
planning, and support functions. TcJ guide and support this contract k RSA 
established an internal MIS Work Group^ to identify the data requiremetrt^ 
for the system. The Work Group identified coverage, efficiency, impact, and 
compliance as the key areas on which information was. needed. In reviewing 
the relationship of. the new standards to RSA's information needs, the Work 
Group recommenderd that the standards and their data elements form the core 
basis for information on the program^'s level of' coverage, efficiency, and 
impact. Other program and exogenous data, and program compliance procedures 
were added as- supporting MIS elements needed by RSA and its component state 
agencies^ Particular design problems — 'such as the recommendations for 
changes in client data items,' for formatting of data elements, and for en- 
suring timeliness of data reporting — were also addressed in this design 




Comprehensive Management Information System for the State/Federal 
Vocational Rehabilitation Program, Final Phase I Rejport /Executive Summary, 
November 1980. Abt Associates, Cambridge, Massachusetts, J^ovember 1980. 

?Cole, Charles B., et. al., Informatio.n Needs Assessment , Abt 
Associates, Cambridge, Massachusetts, September 24, 1980 (Revised). 
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phase. For example, some VR agency directors are worried that sole use 
of placement outcome measures as perfo*rmance indicators might weaken the 
agencies' ability to demonstrate worth, vis-a-vis other social programs. ' 
From their perspective, the program would 'Tiave a stronger hand'» if the MIS-^ 
included a broader set of measures showing "intermediate achievements" by 
clients in terms of gains in life status or functional abilities. This 
concern has arisen due to the 1973 Act»s mandate to serve the severely dis- 
abled, and from the emphasis on Independent Living in the 1978 Act. In 
response to this concern, the Work Group recommended that a pretest be con.-' 
ducted to determine the feasibility of including measures- of non-vocationaX 
benefits (e.g., self-care, housing situation, mobility) within the overall 
MIS. This pretest will occur in FY 1982 and, if successful, will result in 
MIS elements which more fuUy captures the information needs of program 
managers. { / , 

The debate surrounding the issue of non-vocational benefits presages / 
a possible change in the basic philosophy of vocational rehabilitation. If 
the leaders of the program decide in fact to pursue this direction in their 
future legislation and regulations, it will call for additional goals reflected 
in the standards and their data elements. For the time being, however, the' 
stai^dfards express the explicit aims of the program. ^And, while the MIS 
designers ^are experimenting with the implications of these newer thoughts.^. ^ 
related t(5 credit for performance , J the MIS design itself still centers on^ the 
traditional emphasis expresse<t'Tji the revised VR Program Standards. 

CONCLUSION . , 

The rehabilitation system has traditionally benefitted from its ability 
to describe its achievement in terms of a single outcome measure,, the "26**. - 
However, this measure does not differentiate between outcomes of different 
quality, or allow for more difficult clients. Consequently, in response to 
a Congressional directive to develop performance measures^ RSA has developed 
a system of standards which seeks to measure performance over sev-eral pr)?gram 
dimensions. In response, a system was hastily developed, and subsequent 
developmental efforts have resulted in a new system now ready. for implementation, 
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This developmental effort has taken several years and involved state adminis- 
trators, RSA officials, and other rehabilitation experts in a careful anamination 
of the goals of the program and the processes of rehabilitation. The resulting 
measures stress achievement of competitive employment and other quality outcomes, 
cost-effectiveness, and coverage; and for the most part relegate process and 
input concerns to case review and other compliance procedures, and to supportive^ 
analyses. 

The standards have undergone extensive development and refinement since 
their genesis in the 1973 Act. However, a major task remains in terms of 
translating the concept and theory of standards into a functional system which 
fulfills the goals originally motivating ^SA's standards development efforts: 
to provide a vehicle by which RSA and state agencies could assess and analyze 
performance, and direct the program towards improved performance. UntiL 
recently, efforts in ti^e development of standards have focused on identifying 
the appropriate aspects of performance needing measurement (f". e., on choosing 
standards), and o(i identifying data elements to address these various performance 
aspects. Only recently have attempts been made to illustrate how the informa- 
tioiiy^l5rovided by the standards can be used by program managers. The 1978 
Amendments to the 1973 Act deleted the requirement for the evaluation standards. 
^ RSA, however, has continued to support standards development as an evaluation 
activity. It is quite clear, however, -- from the experiences in state agencies 
in the first round of standards development and in the pretest -- that without 
some demonstration of the tie between the standards and anything that matters 
to administrators and counselors m their day-to-day work, the standards will 
form the basis for a system of descriptive statistics and nathing more. RSA 
has had, and continues to have, better program information than many social 
programs; the reporting and analysis of the standards adds to that capacity. 
The real test, however, is whether the standards contribute to performance. 
Through the MIS, the information requirements of the standards can be handled 
more efficiently than is possible in current reporting. But only through 
management's use of the standards that is, through identification of 
p'Irformance problems and ana.Iysis leading to corrective action -- will the 
intended program guidance and program improvements come to be* ' • 

As part of BPA's activities for the standards pretest, we developed 
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a '•paradigm** Ci«e., model) for management use of the^ standards system. 
In the next section we turn to the topic of program managers' use of 
data in guiding their programs. First, we discuss our perceptions of the 
curren t__ 'Is t_a t_e_-QfriJie„-^^ t " ; how. an d t o what extent managers use program 
data in general, and standards data in particular. (These impressions were 
garnered from interviews with administrative personnel in the six MEU's, 
who 'were visited on-site during the pretest.) Following this, we next 
provide a summary description of our proposed paradigm. We hope that,' with 
this summary, the potential benefits deriving from a functioning, effective 
standards system will be made clear-. 
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II* THE USE OF PROGRAM STANDARDS INFOmTION BY 
VR PROGRAM, MANAGEMENT PERSONNEL: 
IMI^feESSIONS ON THE CURRENT STATE OF THE ART, 
AND A PRlyoSED PLAN FOR IMPROVED UTILIZATION OF STANDARDS DATA 

A standards system in VR should facilitate evaluation and management of 
the program, with the goal of improving program performance w However, as was 
discussed in the previous chapter, investigations BPA during the 1976-1978 
standards revision^^tivities indi^fated that, at least in the early years, 
the standards had failed to ImpactLerw agencV management or, by extension, 
on agenc/ performance* During the protest, ^BPA has developed a model for 
use of th'e complete standards by VR program managers to improve agencies' i 
performance* This model, which is presented in detail ;in the Analytic Para- 
digm for the VR Program Standards , iHustrates the "cyclical" nature BPA 
recommends for operation of the complete statndards sys^m/ ' Thus it shows how 
the standards data will be reported, analyzed by evalu^ion and management 
staff , and "fed back" in the form of policy, proceddjf^l, and/or programmatic 
changes for the next operating cycle --to improvp program performance. 

This section addresses the general issue of how program performance 
data — from the standards and from other reporting sources is currently 
used by management personnel in the VR field to jrerform their management 
functions. First, we present an "update" on the current uses of ^program data 
by state agency administrative staff. As part of our pretext activities, 
- BPA staff spoke with top-level administrators in eac^of ' the six Model 
Evaluation Unit -agencies (MEU*s). We explored whether the current system 
of nine program standards had gained in importance as a management tool, 
since our last visits to state agencies in 1976. As well, we wanted to ge^^ ^ 
the administrators' perceptions on how program data is and could be used 
for. managemfent purposes, and on hpw a revised, overall program standards 



^^Berkeley Planning Associates, November, 1981 < 
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system shoiild be designed tq, assure maxiraum impact on agency performance. 

Many issues and_recontmendations were put forth during these discussions, 
and BPA has used -this professional input in our design of a model for 
operation of the standards system. 

After preg[enting our findings from the discussions with administrative 
staff, we present a brief summary of the various components of our overall 
system design. With this, we hope the reader will gain a genej^l understanding 
of how the revised standards system will operate and be used to enhance the 
program's capacity for improved performance. 

4 

\ - 

USING RSA PERFTJRMANCE STANDARDS IN PROGRAM MANAGEMENT: THE CURRENT STATE OF 
THE ART * \ 

This discussion presents a synthesis of the comments received from top- 
" level state agency administrative staff regarding the current and revised 
-Standards. Our interviews in the six MEU state agencies touched on such topics 
as;, current uses of standards data, current uses of data in managing programs, 
^potential utility of the revised standard^ system, design issues for a per- 
formance standards system, arid feasibility of standards implementation. These 
interviews took place in November and December 1980. The consensus was that, 
the^reyi^ed standards system could be useful for both internal management 
and "external 'wJ.ations , if certain design -criteria are met. 

We found little change in terms of the importance attached by administra- 
tors to the current nine standards. In general, administrators continued to 
view the standards activities primarily as a, reporting requirement and not as 
a management tool. Many of the individual data elements were considered 
^ important by administrators. However, delay in data reporting from RSA 
greatly reduced the usefulness of the standards reports for day-to-day manage- 
ment.^ 

Given their/ experiences, ^ it was not surprising that our administrator 

respondents reacted cautiously to the concept of new revisions to the federal 

evaluation set of standards. However, BPA's. specif ic standards and data 

element recommendations were received quite positively by the administrative 

t 

staff. Each o'f the six MEU state agencies already use many of the proposed 

^Our interviews took place in December^ 1980, before the distribution of 
the 1979 Standards Reports. The 1979 data was available in the summer of 1981 
the delay thus has been reduced to 1-1/2 years. 
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data elements in their existing management information systems • The most 
conilnonly used data elements, according to the administrative staff, include: /' 

• percent of closurjgs rehabilitated (% 26) > 

• expenditures per 26 closure; 

• percent of 26 closures obtaining competitive employment; and 

• percent of 26 closures earning ,the minimum wage. 

In addition, most of the other standards data elements are used in at 
least one of the agencies we visited, for purposes of monitoring counselor 
and agency performance* Furth^er, for the most part, MEU administrative ^ 
staff felt that it would be good to have access to^the information provided 
by each of the data elements proposed in the revised standards. The only 
data elements which were seen as problematic were: 

• the benefit-cost ratio and the total net benefit data elements 

* 

(Standard 2). The major problem perceived here is that the 
models appear too complex t^ be ^ use in managing agency 
operations. However, the data elements are seen as powerful 
selling' points for tl>e agency, presuming the agency "succeeds" in 
terms of the two measures. a *. • 

• the causality data elements (standard 6). The major objection 
raised here concerned the validity- of control grjoups used to 
determine causality. The preference (and practice) of each of ^the 
state agencies is to determine causality on a client -by -client - 
basis; that is, by conducting case reviews, with a special eye 
towards quality and appropriateness of services. ^ 

• the satisfaction data elements (standard 8). These data elements, 
while seen as useful for public relations, are not seen as useful 
for managing the agency, primarily because of the subjectivity of 
client appraisals. 

The six stat^ agencies seem to vary in the extent to which administrative 
personnel use data in managing their programs. ^ This is partly due to 
differences in their data-processing resources (e.g., hardware, programmers)* 
However, even in those agencies with sophisticated data processing capabilities 
it appears that program managers have yet to make full use of available data. 
To be sure, data on clients, services, and outcomes are collected and reported 
to program managers. Also, as we alludecl above, data is used for monitoring 



counselor performance and iii "establishing perfprmanc/s goals for counselors. 
However, program data are rarely used in analyzing^ %nd explaining performance, 
gogd^ or poor. Given this,. it would se^m logical to assume that program data 
i^ hot used to manage the program, or to inform program change. In fact, - 
according to bot^h administrative and\evaluation staff , . programs are maliaged 
ipf a reactive fashion rather than in a prospective fa^ion > %n particular, top 
management's prime concern is wit;h reacting to external actors rather than 
focussing, on interi\al management. According to evaluation staff, the majority 
of "special analyses" "(i.e. , one^-time, non-recurring studies) are conducted 
,in reaction to queries pr complaints from external actoi^s (e.g., legislatures, 
consumers)^. In contract, we found little evidence of such internal' management 
techniques as differential performance-based resource allocation systems for 
districts or counselors, or of special studies intended to explain and improve 
performance. ^ ^ - 

During our interviews, hovi^ever,- pi;ogram managertient staff expressed 
interest in having access to* a d^ta analysis system useful for managing their 
programs. They saw t^iree primary benefits from such a system. First, as might 
be expected in the current fiscal climate, administrators felt than an effective 
data, analysis system could help to improve the cost-effectiveness of services ♦ 
Given adequate analysis resources, the standards data elements are seen as one 
vejncle for identifying and investigating problem performance , and for , acting 
on that information to improve cost-effectiveness: through funds reallocation, 
special program development, program and procedural changes, technical assist- 
ance to field staff, and so, on. Secoi^d, administrative staff saw the revised 
s^tandards as useful for selling the program to funding bodies and consumers , 
with a special focus on^measures of cost-effectiveness. This benefit will be 
enhanced, of course, to th^^extent that the agency achieves exemplary perfor- 
mance. However, Recording to^our respondents, some level of benefit is obtained 
simply by having data whefe none existed before (in a useable form). Finally, 
administrative staff expressed the idea that, in general, if designed correctly* 
the standards could lead to better decision-making by administrative staff ; 
that is, to better informed and more effective program management. 

Although our administrative staff respondeiyts perceived some potential 
benefits from a. data analysis system based on the standards, their support 



was qualified by two design concerns. First, to be useful, the information 
must be timely ; that is, .performance should be reported t>n a frequent basis. 
Monthly or quarterly reporting periods were considered ideal, *with six-month 
petiods being termed the "absolute longest** period acceptable. Beyond this 
time-frame, administrators felt^tlie standaTSs ^ataTwouldT.^ ~ 
for management. The current practice of establ ishing performance leads 
based on comparison of last year's national statistics has severely limited the 
use of performance data for management purposes. Analysis of closure and case 
data on a* treatment basis should be a part of the design. Second^ if the 
information is to have an impact on management, it must be accompanied by a 
system for analyzing the data at the micro-level ; that is, in terms of districts 
counselor, and clients, as well as overall state performance. Without such a 
system, program managers would obtain little more from the RSA system than they 
already have: a data system for reporting state performance, but lacking a 
mechanism for investigating causes of unacceptable performance. This concern 
is answered below > where we discuss methods for conducting analysis (termed 
"supportive evaluation**) for investigating and correcting problem performance, 
using a data-based decision support system for state agencies. 

Our discussions also touched on the issue of RSA's use of the standards. 
Several ideas emerged from these discussions. Our administrative respondents 
felt that RSA conceivably use the standards as an incentive to improved 

state agency performance. However, for this to happen,, the administrators felt 
that certain design criteria must be met. First, RSA must actually providfe some 
type of incentive mechanism which will make the stand;ards a salient factor in 
program managers* strategie,s. Maximum incentive impact would be achieved by 
linking RSA funding allocations to the agency '-s standards performance , according 
to administrators. Without such an incentive^ the standards remain just another 
reporting requirement. However, the staff were quick ,tb; note that their pre- 
ference is for positive incentives (i.e., rewards of * suppl^ihental funding) given 
for achieving improved performance, rather than for punitive sanctions (i.e., 
funding cuts) due to failure tp improve. If such incentives- ii^ere established, 
states would have in interest in internalizing the. measures for their *owfi staff 
performance assessments. . • - . 

Beyond the incentive i3sue, the staff noted other concerns they hoped 
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would be addressed prior to standards implementation, particularly if RSA 
intends to make comparisons across states when evaluating agency performance. 
To summarize, the concerns are: • ^ 

• Any measure chosen as a data element mu6t address a valid ^ 

program goaly and agencies "Tnust~'be~capable--o f influe Ti^iqg^ r ' 

performance on those measures • * 

RSA must establish consistent definitions for use by all agencies^ ( 
when reporting performance,^^ Of particular concern are the definitions 
used to classify costs, expenditures, similar benefits, and closure 
types, . 

• RSA must develop some method for prioritizing the conflicting goals 
represented by various standards jdata elements. For example, 

the standards mandate state agencies to maximize coverage (e,g,, 
number served, percent served who are severely disabled) while at 
the same time mandating maximum impact (e,g. percent rehabilitated 
and competitively employed). These conflicts were s.een as the major ^ 
problem needing resolution, before use of an^^ilfcentive system. 

Finally, we discussed the feasibility of obtaining acceptance of the"* • 
standards by field service personnel. In general, our administrative respon- 
dents felt that such a system could be accepted as a monitoring mechanism 

* ^ 

by field staff, although *'it would take some time" and '*of course counselors 
will complain at first'^ States with counselor unions might experience 
greater difficulty in instituting such a system. Also, our respondents 
cautioned that, to avoid merely a ''numbers game'*, the standards would have to: 
CI) reconcile conflicting mandates (e.g. through priorities on, or weighting 
of data ^lements) ; and (2) appear salient to program managers. ThiS second 
, condition wiil be facilitated if state agencies perceive that an incentive 
exists. However, even in the absence of an RSA incentive, program managers 
still might implement a standards-type evaluation system, provided that 
a system exists for using the data at a micro level. 

* To summarize, overall, the administrative personnels we interviewed 
expressed positive, if guarded comments on the revised' standards . State 
agencies" already use many of the proposed data elements in their management 
or monitoring systems, although our impression is that program managers have 
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yet to make full .use even of the data they have. Current use tends i:o be 
reactive iiTnature: information dissemination to external actors, rather 
than information utilized inlj^lially for program change and improvement. 
However, in the current fiscal climate, administrators' interest in such 
'^data uses seems*^to be heightened. Much concern exists over whether RSA 
would use the standards to compare state.s and, if so, whether RSA will ensure 
consistent definitions and reporting procedu:i^es for all agencies. Finally, 
given time, they felt that the standards would be accepted by counselors and 
other field personnel . ' * * - 

A hiODEL FOR MANAGEMENT USE OF THE REVISED STANDARDS DATA 

When fully implemented, the standards design will comprise a useful 
system for VR program evaluation and management. So that the reader may 
understand how the standards, data elements, and data collection forms 
discussed herein> will fit into this sytem in practice, we would like first 
to briefly discuss the various components of the total system design, and 
to illustrate how tHese components fit together into one integrated system 
for using the st^andards information. A detailed presentation of the system 
appears in our report on the analytic paradigm for the program standards. 
Here we sketch out the overall context, so that the larger picture will be 
evident. 

The components of the Program Standards' System are as^ follows: 

• a specification of the goals and functions of the VR program; 

• a set of standards data elements to measure the program's performance 
and procedural goals; 

• a process for setting performance expectations on these data elements 

• a model for decision-support systems which allows managers to use . 
program data to help investigate possible causes of problematic 
performance; and 

• a reporting system for the standards. 

The Goals and Functions of the VR Pi^ogram 

RSA has issued a variety of goals over the course of its existence. 

• / 
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TTiese goals have appeared in such publications as the RSA Fo«*ward Plan, 
short- and long-range plans >^ Information Memoranda, ^d so on. The goals 
have addressed the full range of activities, concerns and foci of the 
rehabilitation field, including: 

• facilities grants and management; ' , * /.J- rf * 

• general advocacy and policy-development activities; 
similar benefits and inter-agency cooperaticffT^" ^ 
rehabilitation research; - ^ . 



• . independent living; 



management c^^jiabilities ; and, most relevant for our purposes, 
service delivery, outcomes, and impact. 



IVhile RSA has not lacked for goal statements, it has recently received 
criticism concerning the ef fectivehess of its system for implementing its 
own goals. Some of the specific criticisms -allege that RSA has: 

• f-ailed to operational ize its goals (i*e., specify them, in measurable 
terms); and 

• failed to have a system for transmitting performance information 
to the appropriate program managers for use in decision makiiig. 

In .response to these criticism, in 1980, the acting RSA commissioner 

2 

issued the following set dr. broajl goals for the agency: 

• insuring compliance with the legislation , and 

• encouraging and supporting State agency efforts to improve 
the quality and cost-ef f ectiveness ' of the program according 
to nationally defined priorities. 

These two statements are obviously quite broad, and require some elu- 
cidation as to their meaning in the context of the basic VR program. In 
the Analytic Paradigm report we discuss how these goals translate into more 
specific subgoals (i,e, , performance objectives) relevant to the basic VR 
.program. In particular,-l_we_dllustrate how the-revised performance and ' . 



A description of RSA's management and information processes appears 
in the Evaluability Assessment of the^Role of the Rehabilitation Services 
Administration in Managing the Basic VR Program by Urban Systems Research 
and Engineering, Inc. (1980) ' 

2 

RSA Memo, '^Development of the RSA FY81 Operational Plan,** August 29, 
1980^ 
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procedural standard's relate to the program's, goals, and how the standards 
data elements provide operational measures of the program's success in achiev- 
ing, its goals. 



Set of Data Elements 



^ The set of data^eleinents used to measure the performance goals are 
presented elsewhejre in this report, along with discussions of their validity 
and reliability, ^Here it should be noted what these data elements do and do 
not measure. Specifically, the program standards measure a state's: 

• level o£f^verage , i.e., the proportion of the eligible 
population being served; 

• impact of its service to clients; 

• efficiency in serving clients; and 

• attention to key procedures , 
IVhat are not included are measures of: 

• inputs , e.g. what kind o? VR eounselors are hired, what kinds 
of services are prescribed; 

• systems impacts, e.g. on attitudes of employees toxvard the ^ 
disabled, on referral networks in the Community; and 

• financial operations , e.g: budgetary systems. 

Thus, the program standards data elements are mostly oriented to' measuring 
the performance of VR agencies in .helping individual clients; they leave 
decisions of how to achieve these performance go^ls to individual agencies 
and VR counselors J 

Process for Setting Performance Expectations 



JVhat is very different about this standards system is that it is a 
prospective system designed to change the performance of VR agencies 
through prior specification of performance expectations, rather than a , 
retrospective system that just reports on past performance as compared to 
other states. There are many options for setting these performance expecta- 
^tiofis, including:* 

using average performance of all agencies, or some subset of agenties 
(e.g., in the region); 
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• specifying comparable agencies or sister agencies with exemplary 
performance and using their performance as an expectatioiC; . 

, • using norms of performance set by Congress, RSA, CSAVR, etc.; 

• setting minimum (rather than maximum) kinds of performance the same 
way; . . 

• drawing on a particular agency's past performance to set limits^ 
on future performance; and * . * ' 

• negotiating witft a particular state agency to.-set a realistic 
expectation, perhaps through the Regional Officer. 

The existing standards draw on central tendency statistics to judge 
whether a state program performed adequately in the past /ear. This approach 
implicitly assumes that performance. should have some kind of normal distribu- 
tion with some states always being poor and other states being extraordinarily 
successful. This assumption is clearly unfounded, however. It could well be 
that all states have an acceptable performance, or that all states are perform 

f ing poorly in terms of some standards. The central tendenc^statistical 
approach, while descriptive, does not examine the level of typical performance 
with what is reasonable or desirable, but instead automatically generates 
"failures^^ and ''successes'^ among state programs. The more that state agenciojS 
are fairly similar in performance in terms of some data element, the more \ 
arbitr^y the central tendency approach becomes. Finally, because the central 
tendency approach. requires the data for all state programs to be available so 

/that the distribution could be calculated, performance "norms" for state 
programs are dependent on the timeliness of the last state's submission of 
data. This. greatly delayed the availability of the evaluation data and made 
ilse of the standards infeasible for the ongoing monitoring of the program 
performance during the yearr Even with improved measures of central tendency, 

Vthese criticism hold true. We-rec'Smmend a performance level procedure where 
the central tendency approach becomes merely one of several methods which can 
be used to set performance goals , rather than being' the sole method* Further^ 
we recommend an additional, aijid even more significant change in the process 
by which performance goals are set: rather than continuing with the post-hoc 
system based on^^tional normsj, we recommend placing responsibility within ^ 
each state to set its^^ o\m objectives for the level of performance to be 
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achieved during an upcoming fiscal year, for eaqh data element • IVhen setting 
performance objectives, state agencies might be anticipated to look at their 
past performance, at the levels of performance being achieved by other state 
programs that agency staff view as comparable, at the performance nation-wide,, 
and at pending changes in state economic conditions, policies on client and 
service mix, and other unique "state factors which might affect perfonnance. 
RSA might provide technical assistance to the state agency in identifying 
appropriate levels, and participate in the state's setting of its goals, but 
the lead and principal responsibility in setting objectives for performance 
for the coming fiscal year would be with the state agency. The new system 
recognizes that state agencies best understand the needs of their programs, 
that there are appropriate differences among state agencies in* policy priorities, 
and that it is the state agency which must accept that there are performance 
problems or shortfalls if needed improvements are to be identified and imple- 
mented. 

A particular advantage of this reliance on state agencies to set perfor- 
mance level objectives is that it permits the standards system to be used for 
moriitoring and assessing the ongoing program. State agencies can use their ^ 
in-Hoaw-data systems to monitor individual data "items on a monthly or quarterly 
basis, and to see if the program is on target in terms of moving toward annual* 
goals or sustaining acceptable rates of quality closures. Thus^ the standards 
evaluaty^on .system can provide much more immediate' feedbacl^to program manage- 
ment to lead to improvements, in performance. Th5 existingl central tendency- 
based system results in feedback years after the program conduct giving rise 
to performance problems, and is useful principally in spotting overall problems 
needing the attention of long-range , planning ahd systems improvement. The 
proposed standards system can be used much more easily for ongoing program 
management and improvement. • . ^ ^ 

The Reporting Sji^stem . ^ 

The role of the reporting system in the standards system is to provide 
the vehicle for bringing together the various sources of data so that a 
particular agency's attainment for a specific time period can be compared to 
its obj,ectives for the perrod. In addition, the reporting system must provide 
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program managers with the capability to investigate problematic performance, 
as we shall describe subsequently • To do these two things, a reporting system 
has been designed to: • ^ 

• specify how data collection for the standards is- to be performed; 
when new data collection is needed , T7g • , OTT^eamings at follow-up, 

• bring together th4 various types of data from the various sources, 
e.g., from the MIS, from the financial reporting systems; 
keep track of past performance as well as current expectations; 
present the findings in an easy to use, easy to understand 

way, without unwieldy reports, emphasizing graphical presentations 
as well as plain numbers; and 

• make sure that the reporting of rissults occurs in a timely fashioa, 
so that future performance can be affected our recommendation is 
that repctrtmg be done quarterly. 

The new data collection for the standards system is described later in this 
report, as are the various sources of data ^or tWe standards. The procedures 
and sources have been* integrated with the Management Information System (MIS) 
being developed separately by RSA and Abt Associates. Indeed, the refined 
standards are central to the MIS sirtce they focus on the program outcomes and 
achievement of overall program goals. The MIS, while obviously serving many 
additional objectives as- well, will provide information readily to state ageri'cies 
that would show p^erformance in terms of the standards and also be usable in 
identifying how to improve performance. However, ev^n if the MIS were to be 
unexpectedly delayed in implementation, the standards sVstem is compatible with 
the kinds of R-500 data compilations routjjiely generated even now in many state 
agencies' internal information systems. Thus, the evaluation standards system 
could be adapted by individual state agencies^for their use, independently of 
RSA's implementation of the MIS., 4 ^ . 

Supportive Evaluation System: Decision Support 



Out of the s.tandards ,teporting system will come the clear indication 
that some agencies on some- .data elements will not have met tlfeir objectives 
for level of attainment, ^e standards system ^does not stop there, however ^ 
but instead moves to investigate the reasons for problematic attainment and 
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to develop corrective actions as part of a data based decision support system. 

The elaboration of this system has been a major effort of the pretest and 

re^finement effort. Chapter VI includes a discussion and example bf this system. 
« 

The purpose, of this system is to close the gap between r^pt>rting on the standards 

and "actions based on the standards. Thus, the system must provide timely results, 

utilize existing information (such as the standards system, the MIS) as much as 

possible, be understandable to the key actors (not just the evaluation research 

staff), and be flexible in dealing with new problems. When in place, the 

decision support system will: 

• provide an ability to pinpoint causes for problems in attainment; and 

' identify strategies leading to enhanced attainment.^ 

p 

The decision model- for the program standards system specifies how 

> 

the information obtained from the reporting system can be translated into . 
actions by various levels of program management. One might mistakenly assume^ 
that, since the Standards have been developed by RSA, the standards information 
system is of use only to RSA. To speak plainly, nothing could be further from . 
the truth. First, since the^ basic VR program is a state-federal partnership, 
the goals set out for the federal agency apply to the rehabilitation programs 
operated by states. Second, although we anticipate that RSA will initially 
take lead responsibility for analyses and decisioi^-making on the basis of* 
the standards' system, it is neither intended nor desired that states be 
frozen out of the processes embodied within the standards system, nor that ' 
the system should have little relevance or use by states in their management • 
processes. In fact, as we have noted in this and earlier chapters, we can 
expect little impact on state agency performance from the standards if state 
managers perceive the standards system as irrelevant to their management 
functions. By developing a decision model for use by state managers, the 
desired link between information and action is facilitated. As discussed 
earlier, xhe need for an effective paradigm for using the standards information 
in manai^ijig the^VR program was made very clear in BPA's visits to states 
during the early phase of standards development^ The problem, it appeared, 
u'ds that implementation of the standards did not include a paradigm for their 
j^use.' In particular, no paradigm was developed which contained all the elements 
nccessary^'to link performance on the standards* to improved manageability. 
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Thus, while the standards have been used fpr ie.scriptive reports, they have not 
in general served as an incentive or aid to improved performance, particularly 
within state agencies. . # . - - 

Our purpose in the analytic paradigm is to design a comprehensive system 
for analping information obtained from the standards systejn, ^tid for identify 



ing areas* for action to encourage program improvement. . 

The new standards evaluation system promises to solve a number of 
'long-standing rehabilitation program needs: it provides an evaluation 
framework for monitoring performance based on program outcome^T"^^ 
the leadership ^ble for 'deciding how to improve. performance with the state 
agei)cy; it provides a workable decision support system linked with an agency 
MIS and structured to enhance use by management to improVe the progra:m; it 
provides a clear but limited role for RSA as the source of technical assis- ^ 
tance to the stages, the developer of basic systems technology for helping 
states in evaluation, ancl the periodic conduct of special evaluation studies 
that would be infeasible or' :jLnef f iciently mounted by a s:ingle state* 

The analytic paradigm represents the broader context into which the ^ 
Program Standards and data elements will fit. We turn next to the detailed 
discussions of the Performance Standards and .d?Lta elements. -/ 



1 X * " . i 

The complete system design is presented in our separate Analyt.iiC 

Paradigm for the Revis^jd VR Program Standards, available from RSA. 
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IV. THE EIGHT PROGRAM .PERFORMANCE STANDARDS 

INTRODUCTION 

This section discusses each of the eight performance standard^ designee 
to measure the achiJ^vement of a desired outcome or ^al of vocational rehab 
ilitation. The standards addres's several pivotal issues of VJ^, Wese are 
primarily: / ^ 

The -extent to which VR serves the target population; and 

The value of VR services, 'both to society and- to t;he disabled 

client, including 

maximisation of 26 closures; increased economic indepen- 
dence, and retention of earnings or improve,ment in gainful 
activity; 

client satisfaction; * 
demonstration of a causal relationsl(ip bet\>[ben VR services 
y and client outcomes; and ! \\ 

y - * ' 

demonstration o£ the cost-bene-^it and cost-ef'fectiveness 

• ^ ^ - ' ' ; V' ^ 

of VR services. \- 

In' this sect ioVi^ each of the eight standards is presented in detail. 
The discussion is organised into eight sub-sections (one for each standard) 
and includes the rationale for each standard and data element, the formulas 
for each data element, and the sources from which data will be collected. 
Th'e comments from state agencies and Model Evaluation Units regarding each 
standard and data element are summarized and discussed. Following our res- 
ponse, to reviewers* comments', we present our preliminary re;commendations . for 
each standard and d^ta ei^^^nt. Finally, at the end o.f each of the eight 
sub-sections we present a brief analysis of the data element statistics col- 
lected from each of the six stateJ^articipating in the pretest. 
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STANDARD 1: VOCATIONAL REHABILITATION SHALL SERVE THE MAXIMUM PROPORTION 
OF THE POTENTIALLY ELIGIBLE TARGET POPULATION^ SUBJECT TO 
THE LEVEL OF FEDERAL PROGRAM FUNDING AND PRIORITIES AMONG 
CLIENTS. 

Data Elements: (i) Comparison of caseload served to expenditures 
(ii) Clients served per 100,000 population 

This standard addresses coverage, or the extent to which the voca- 
tional rehabilitation program is serving the' eligible target population. 
The ;ieed to ensure accessibility of servic^ to all the eligible disabled 
is o.f j)aramount importance to RSA and the' states. Given this standard's 
focus, we feel that it fits in well with the cost-effectiveness goal. On 
its fa<;a, the standard is concerned with the '^effectiveness" aspect of 
the cost-effectiveness questions: increased service .coverage of the eli- 
gibly population is one indication of incre.ased effectiveness on the part 
of a state agency. In fict, coverage irepresents one of the most basic 
aspects of aerogram's effectiveness. Alone, this standard ignores 
considerations of the quality of the coverage *(i .e. , the appropriateness 
and utility of the program's activities in the clients' behalf, ^^d the . 
clients' service outcomes). However, these considerations are addressed, 
by other standards. • 

The comments and suggestions of state agencies on the standard can b.e 
summarized as follows: 

# The standard does not take into account the proportion of severely 
disabled in a state. Federal "priorities" are mentioned in the 

0 standard, but not in the\data» element. 

• This standard can be used for conj)arisons within a state over 
time (e.gv, for purposes of internal management), but it cannot 
be used for comparisons across ^tates. A weighting system ds 
needed for cross-state comparison. 
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• We should iQok at the number served rather than the proportion 
/served. . ' " 

■ ^ 

• There is tieither a good definition of "target population" nor a 
, good methodology for identifying the target population once it 

is defined. 

■ • y \ ■ 

The issue of severe disability is mentioned m connection with several 
standards and data elements. The usual suggestion is that the data element 
be adjusted for percent severe. Such an adjustment would be appropriate in 
relation to certain data elements, but may be less appropriate for other 
data elements. For example, the proportion of severely disabled within a 
caseload can reasonably be expected to impact negatively on a state's total 
case volume (i-e,, caseload size) and on its costs. Severely disabled 
clients, a group , could be expected to require a longer time to complete 
rehabilitation services, and to require a greater amount of counselors* 
time, than Kould be necessary for the group of non-severely disabled clients 

\As a result, when the severely disabled increase as a proportion of total 
caseload, time-in-process should increase and counselors' caseload capacity 
should decrease. Both of these factors point ^o a decrease in the state's 
caseload volume potential; that is, to decreases in coverage. Further, costs 
per client might be 'expected to go up, both because of increased counselor 
time requirements per client, and because the severely disabled may require 
more extensive and more expensive services than the non-severely disabled, 
in order to achieve rehabilitation. Given the above , arguments, adjustments 
for percent severely disabled may be appropriate for data elements related | 
to coverage and cost-effectiveness. - 

Such adjustments may4jot be appropriate, however, for data elements 
concerned with service impact. In^a national study conducted by Berkeley 
Planning **Associates on VR clients closed in 1975, we found that being 
"severely disabled" had no statistically significant effect on a v^iety 
► of impact inditators; including successfi^l rehabilitation (closed 26), 
aphicvin^ competitive employment, earning the minimum , wage , or retaining 
Earnings one year after closure. Tl\at is, other things equal, severely 



^Berkeley Planning Associates; 'Implementing the Rehabilitation Act 
of 1975>: The VR Program Response; February, 1978. 
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disabled clients were no less likely (in terms of statistical probaTjility) 

/ ' V ^ 

to experience these favorable outcomes than were non-severely disabled 
Clients, IVhile these findings may seem contradictory to common sense, we 
/can think of at least two reasoris why they could result. First, the 

/ definition of "severe disabiiitr* is quite broad. A client ^ 
'may be labelled "severely disabled" for one of many reasons, not always 

relat^Jd to the client's specific disability (for example, all clients receiving 
SSDI are by definition "severely disabled"). Further, it is even 
questionable whether some disabilities included in the definition really 
constitute a severe handicap to employment, at least in all cases. For 
exang)le, while blindness and deafness both of which are included in the 
SD definition,-- could be termed severe (i.e., major), disabilities , they - 
may not, at least in all cases, constitute a severe handicap to employment . 
It follows, then, that they may not constitute a severe barrier to rehabili- 
tation or to other favorable vocational outcomes. In short, all things 
considered, the definition of severe disability probably fails as a dimension 
on which to distinguish clients in terms of their probability for successful 
outcomes . s 

There is a second reason why severely disabled clients may be as 
likely to achieve favorable outcomes as the non-severely disabled. Specif i- ^ 
cally, although SD's may require more time and cost, if the extra time and 
cost is committed, then chances for success by the severely disabled client 
are a3 high as for non-severely disabled clients. Stated differently, the 
"technology" for rehabilitating SD clients exists. To the extent that 
counselors are willing and able to commit the resources necessary, then 
SD's are as likely to succeed as non-SD's. 

Given the above arguments, we feel that it probably is inappropriate 
to adjust performance on impact-related data ^elements for states' .levels 
of service to the severely disabled. We would make one caveat, however. 
Specifically, our study, previously quoted, looked at a gifoup of clients 
closed relatively soon after the prior^ity service mandate was promulgated. 
It may be that in succeeding years states have attempted to serve an even 
areater number of SD clients who have truly severe handicaps to employment. 
If this is true, then the SD population as a group may indeed have a lower 
probability , for success than non-SD»s, and adjustments to impact data 
elements may be appropriate. , . 
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^ - Given ^he potential valiility and utility of adjustments for percent 
severe, .the question becomes "how should we make the adjustment?" There 
are two alternative ways to deal with this issug/^' Both involve ^the use 
of a -"percent severe" statistic, defined as the ratio' of severely disabled 
' clients accepted for VR to the total population accepted (i.e.-. Status I'O 
and above) in a given year; . - ^ -'^ " : 

I , I # severely disabled clients served in a year 

# total clients served in a year ~ 



This statistic could be presented on^its own either as a new data^^element 
under Standard 1, or as.j secondary statistic in the reporting system. 
We recommend that it be made a data element, given the legislative import- 
ahce attached to serving the severely disabled. However, either alternative 
would allow administrators to take severity into account when evaluating 
performance. Unlike the proposal for directly adjusting performancevleyels 
for percent severe where the effect of the adjustment would be^ 'buried - ^ 
within the computation of a data element either of tfie two alternatives; 
would give administrators considerable discretion in deciding where and to 
what degree it is most appropriat^e to adjust for severity. This method, is 
in keeping wiTth the Standards overal 1 goals of simplicity and flexibility. 

The second criticism non-comparability across states -- is also' 
applicable to other standards. It must be pointed out that such comparisons 
should not be construed as a co^ntest to see which states achieve the highest 
score. Rather , interstate comparison's are for informational purposes to 
help itSA and the state agencies to point out "model"^ agencies and, through 
comparative analysis, to pinpoint where and how performance can b^ improved* 
As such, the most appropriate cross-state comparisons are those bej:ween 
similar states. A weighting system that adjusts for dissimilarities would 
be far too cumbersom^ "tc* be practical. Further, as is discussed in the 
Analytic Paradigm, the major . evaluative concern of the standards i« to 
encourage improved performance by a state in relation to its previous • , 
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performance, fhe pux^ose is not to pit one state against the other. 

The most important comparisons for this and other standards are 
within-state comparison over time. Once states have identified problems 
and devised strategies for their solution, a year-to-year intrastate com- 
parison will give- the states the necessary information to evaluate the 
success of their strategies. Adjustments for state variation are not 
needed for this purpose- 

Regarding the suggestion that we look 'at the number rather than the 
proportion served, we can only say that it would be grossly biased in favor 
of the larger states. Naturally, there would be calls for adjustments based 
on a staters size. Since this adjustment is incorporated into the very 
notion of a proportion, we feel that any change would be unnecessary. 

Finally, we recognize the importance of the "target group" question. 
• -* 
This issue will be addressed under data element (ii^. r 



Recommendations 
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The following are our recommendations for Standard 1. 

# The Standard itself should be retained in its present form, with 
its present wording. • . ^ ' 

• , In order to address the "priorities a^ong clients" mentioned 

in the Standard, a '^percent severe" data^/Aement should be included. 
It should be defined .as follows: 



^ severely disabled clients served in a year 
^ total clients served in a year 



By "served,'* we mean clients in Statuses 10 and ab'Ove. The data for 
this data element would come from the RSA-113 report (after that report 
is implements i) . The particul,ai*'^ta sources are; . 



* severe l5' disablad served 




RSA-115 




IIA4b + IIA5b + IIA6b + 


IIA7b 


Total clients served; 




RSA-113 




IIA4a + nA5a + IIA6a + 


IIA7a 
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DATA ELEM ENT 1 (i): COMPARISON OF CASELOAD SERVED TO EXPENDITURES 
• II ' 

This data element was originally conceived aa an indicator which 
could tell states if they had achieved the- level of coverage they should' 
have, given their level of resources. As such, while this potential 

m 

element does not directly measure an agency's service of clients in terms 
of eligible target population, it dpes provide a type of relative level 
of effort and efficiency measure across. agencies. When the two concepts 
are considered in tandem, this measure 'provides one proxy for coverage • 
The form and data sources f©r the data element are as follows: 



% national caseload served 
% national expenditures 



Data Source 


* Data Item Identification 


■RSA-lfrr^^ ^ • . 


Cases Served = Aggregated states' active 
cases and cases closed (statuses 10-30) 


RSA-2 


National expenditure = Aggregated states' ex- 
penditures for all monies (state and federal) 
spent under cohtrol of VR agency (110, Trust 
Fund, SSI and I§E) 



The comments of state agencies are presented below, 

• If there is a major difference among states in percent severely 
disabled, they should not be compared, 

• We need to make allowances for inter-area differences in price 
levels, 

• We might want to separate administrative costs from other costs. 

• The denominator is based on all statuses, but the numerator 
includes only Status 10 and higher. Choose one or the other 
but be consistent. 
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•* There are differences across states in the proportion of 

total funding provided by the federal government. A low ^ 
proportion will make a state look better. 

• This is not really a ''coverage" data element. 

Some of these criticisms can be answered* quite briefly. First, as 
discussed above, it may indeed be appropriate to either not compare states 
which differ/ in their percent severe, or to adjust for that difference. 
Second, we reject the suggestion of separating administrative costs /rom 
other costs, because of the inconsistencies in accounting procedures across 
states. It would be most ^fficult to accurately separate and account 
for administrative expenditures. Yet*, if costs are s-eparated without 
accurate accounting, we may distort the results. Use of total expenditures 
eliminates this problem. Perhaps even more importantly, the intent of this 
data element is to indicate efficiency in coverage of the eligibile 
population. To the extent that administrative expenditures are inordinately 
high, efficiency in coverage has not been achieved. This should be reflected 
in the data element. 

We are aware of the fact that the numerator and denominator jare not 
based on the same population, but^we do not consider this an inconsistency. 
The mandate of Standard ^ is. to maximize service, not client assessment. If 
an agency spends an inordinate amount of resources assessing clients who 
are eventually closed 08, it indicates some degree' of in^^fx^ency in the 
agency's intake and assessment process. , 

We agree that there are inter-area differences/in price levels which 

should be taken into account; however, the means to do so are currently 

t 

lacking. Since this. criticism is equally applicable to the data elements 
of Standard 2, the problem is more serious. We suggest that RSA develop - 
a regional or state-by-state VR cost index, similar to the regional Consumer 
Price Index already in existence.- This index could be used to adjust ajl 
expenditure figures used in the Standards. • • 

As regards the cbmment on federaj contributions as a proportion of 
total resources, we feel there must have been a misunderstanding about^^ . 
how the data element is computed. Specifically, ^'national expenditures'' 

■ .7 . ■ . ~ . • , 
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is defined, as the .sum of all agencies' total expenditures from both 
state and federal sources. Thus, the extent -of state overmatch is 
irrelevant. , . 

The most teljing criticism, in our opinion, is that this data 
element dbes ^not 'really deal with coverage. Instead, ^t measures the 
efficiency of a state's use of resources. Inasmuch as this issue is 
addressed in Standard 2, its inclusion under Standard 1 is both redttndant 
and inappropriate.^ 

There^is an additional problem with"^ this data element, one not men- 
tioned by, reviewers. The crux of the issue is that this data element 
has problems whi^n used, as a pecrformance measur^across states, because a 
given state can improve only at the expense of others. The form of this 
data element makes it most unlikely that all states simultaneously could 
achieve a minimal-acceptable level of performance (i.e., a ratio of 1.0) • 
This contradicts -our main concern that states improve on their past 
per^formance* 

Recommendations 

Our, recommendations are two-fold: 

• This data element should be dropped from the standards system 
because (1) it does not address ^'coverage'', and 2) it l)as 
problems as a comparative measure of performance. How^fer, 
RSA could continueto con^gute this variable to pinpoint '%tfde'l" 
fi.e., efficient) anra '^problem** (inefficient) agencies, for 
purposes of ocmrpa^tive analysis and triggering o£ technical 

. , a*ssi stance. 

• Ifv the data^element is not droppe,d, a regional or state -by-state ' 
VR co^> ijjdex should be developed/- It should either be used to - 
adjust all expenditure figures or be' reported as a suppl-ementary • 
datu^m^^d in the interpretation of this and oth^r expenditure- 
relate^ dat^fiff^fji^ \ , 
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DATA ELEMENT 1 Cii): CLIENTS SERVED PER lOO.OtiO POPULATIQN ' 



Like potential data element 1 (i),'this data element does not 
provide a true estimate of the level of coverage of eligible target pop- 

\ ulation, it does, however, provide a proxy measure of the size of the target 
population by using the overall state , population /~ If estimating the 

' target population were a straightforward matter, this proxy would not be ^ 
desirable or needed, but given the need for long-term development of 
an acceptable target population measure, this proxy serves a potentially 
useful purpose. Also, it is used now by state agencies, and thus it has 
some management utility and validity as a performance measure. The form 
and data sources for the element are as follows: 



serveg m a given year 
100,000 population 



Data Source 


Data Item Identification 


RSA-101 


Number served = Ag'gregated states* active 
cases and cases closed (statuses 10-30) 


U.S. Census. 
Series P-25 

* 


Current best pro2cy for ''eligible population'' - 
(that is, proxy in relative, not absolute 
sense across states). , Periodic supportive 
evaluation shoul^ update existing and test 
new methodologies developed for estimating' 
eligible population. [ 
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The comments about this data element focused on four basic issues: 

■ i^ ^ 

• The validity of the estimated target population is questionable. 
Wliile using state population (from the Census Bureau^ rather 
than each state 's computation of the target populatjion elimi"?" 
nates an unnecessary bias, it is still not a good estimate of 

the disabled population. Moreover, this da^a^ement is largely , 

useless except right after a census , because of problems in 

developing meaningful estimates of some states' populations 

between census years. It was suggested that RSA work 

with sister agencies to get ^ better estimate of the disabled 

papulation. 

• This data element -reflects the availability of other public 
services in a state more than it^^rp^ects vocational rehabili- 
tation. ' - ' . . ' 

• ''Cases served" should include applicants as well 'as active' cases. 
That is, "cases served" shoul'jd include Status 02 and higher* 

• Adjust for percent severe disabilities served\o avoid creaming. 

The issue of including Status 02 ani higher in^the definition of 
"cases served" was addressed under data element (i^^fiy^ To reiterate, the 
coverage^ mandate of Standard 1 applies to client seiVice, not to client 
assessment. . • . * 

With regard to the severely disabled issue, we.woyld note that this 
data ^element represents one of the cases where percent severe should be 
taken into account when judging performance. 

Insofar as*a VR agency is located»in a state which generously supports 
public health and 'rehabilitation services other than VR, it is a boon to 
that agency. It is, of course, a relevant factor in deciding Which states 
should be considered exemplary models by RSA. States .which are able to 
expand their service capacity through cooperative agreements and use of 
similar. benefits'could be considered "models" for other states, and RSA 
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will want to encourage such expansion of service capacity ♦ To some 
extent, however) this will Be beyond the control of RSA or th^ agency^ and 
the presenc'e or absence of similar bene'^its'^wlr^ be used as a 
reason to reward ,pr punish state agencies. It will, however, be used as 
a potential explanatory variable in determining (i.e., conditioning; ^ 
agencies' levels of performance. Finally, the presence or absence of other 
public services is irrelevant when making year-to-year comparisons within 
a state, which is the primary' use to which the standards will be put. 

The validity of the target population is^ a more serious isjsue. 
There is some merit to the arguments that population estimates between 
census years lack precision for those states experiencing rapid population 
growth or decline, and that in any event, total state population may not be 
the most valid indicator of a state's disabled population. IVhile.not a true 
estimate of the level of service by an agency, this statistic does provide 
a proxy measure of the size af the target population. . To the extent that 
disabilities are uniformly distjyibuted across states, state population, will 
be. an unbiased proxy for the target group. As noted above, were estima- 
ting the target population a straightforward matter, this proxy would not 
be desirable or heeded; however, this proxy can serve a usefu] purpose 
until an acceptable target population measure is developed. 

We agree that RSA should^ become invon^d in an effort to. develop a_^ 
better estimate £f>T the t*arget population. The Ridge-Worral method seems 
to be an appropriate point from which to proceed in this development. Its 

mam failing is that currently the figures it generates are out-of-date 

» 

because they are based on the 1970 census. This can be rectified in the 
short term by using the 1980 census to bring the figures up-to-date. In 
"the long term, it is also necessa:||ij|l^o persuade the Census Bureau to col- 
lect more pertinent information on disabilities when conducting population 
Estimates between census years, so that the Ridge-Wprral ^thod can be used 
m jfuccoeding -years with more current input data. IVhile the Censujs Bureau's 
compliance is by no means guaranteed, this option is certainly worth 'ex- 
ploring. 
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Recommeridat ions - , ' 

We have two recommendations regarding this data element: * * 

• The data' element should be retained ^ is for the short term. 

• RSA should become involved in the develbpment of a more precise 
estimate of the target population for long term use. The Ridg^- 
Worral method should serve as the point of departure in this 
development, and it may suffice if - cooperatTori from the Census 
Bureau can be obtained. ^ 

FINPrNGS FROM THE PRETEST DATA 

Table 8 presents the results of the analysis'*of pretest data col- 
lected for the Standard 1 data elements. We have recommended' dropping 
da'ta^ element (i) for a variety of 'reasons. First, the figures may 
i^eflbct the size, caseload capacity, and attendant economies of scale 
of different agencies, as much as or more than they reflect **eff iciency** 
of coverage" per se. Second', we noted earlier that the data element 
represents a "closed system"; one state advances at the e^cpense of 
another. This is illustrated quite clearly in the table, although the 
data i.s '^static" (i.e., shows only one year's worth of performance). 
Consequently, while the Pennsylvania data shows far more success than 
the other five agencies, careful attention should be given to the con- 
tributing factors not Captured in' this measurement. 

Figures for data element (ii) vary quite substantially. Again, 
Pennsylvania registers the highest figure, Mississippi Blind the lowest. 
However, two considerations should be taken into account when judging 
Mississippi's performance. First, as a blind agency, its relevant popu- 
lation is a much smaller proportion of the state's total population than 
would bo the case for the other^ five general agencies. This suggests 
that, to the extent states are compared, they should be compared within 
types of agencies, as is done within th^ current stan^rds system. 
Second^ its client population is much more likely to contain a very high 
proportion of severely disabled, which as discussed -above would lead to 
a lower performance expectation on this coverage data element. 



' ^ ' Table 8 

' Statistics for Standard 1 



Data niement 


^ Hquation 

<«• 


State 


VA 


PA 

> 


. MI 




■ OR 


.BLINI> • 


•* 

1 (i) ■ ' 


% National' Caseload 
Served^- 
% National Expenditures 


' "3' 
.81 


1.26^ . 


? 

.94 2 


i 


".7.4' . 


' ' 3'*' 
.37 . 

f 


T fii) 

• 


i# Served in Given Year/ 
(1979) - . . • 
■100,000 Population 


474*. 1 


690.9 


% 

■ 416.2 


404.6 


538.3 


05gi4- , 



Data Sources: 



1, RSA-101 • . ... 

2 RSA-2 ' ■ , ' ^ 

3 Based on s^ix-state sample . ^ , 

4 Current Population Report, Population Estimates an4 Projections, July 1579 
U.S. Bureau of the Census; Series P. 25,. //876 ' ' - 
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STANDARD 2: THE VOCATIONAL REHAE^ILltATION PROGRAM SHALL USE RESOURCES 
IN A. COST-EFFECTIVE MANNER AND SHOW A POSITIVE .RETURN TO 
SOCIETY OF INVESTMENT IN VOCATIONAL REHABljLITATIOM OF 
DISABLED CLIENTS. . ' ' . 



Data Elements:' 




(iii) 



Expenditure per competitively* employed closure. 
Expenditure per 26 closure 



DiscotrfrteS net present .value 



Benefit-cost ratio 



This standard. is^ the one most obviously related to the program's ' 
cost-effectiveness go^ . Two issues, are addressed by. this standard. The 
^ first is' the issu^ of cost-effedtivehess : with the' financial resources 
available to the stat^ (or sub-region, or district, .or counselor), how 
success^lly did it - achieve* desired objectives? The second issue- revolver 
around cost -benefit concerns (i.e., ,"retum on investment") • .Specifically, 
the standard, asks the question: Are we getting more out of the program . 
than we 'put in?* , Currently, ^enef its from the program are measured 
primarily in monetary terms (e.g^, in: terms of wages .earned, taxes paid, 
and public assistance, foregone),^ Unfortunately, thi3. 'Iiard-nosed" monetary 
focus omits cons Ider^rt ion of many of the other benefits derived fropi VR 

• * ^ " 

(e.g. /increased functional capacity). Woifk is underway to develop methods 
for takiWg such benefits into account. In the interiln, however, monetary ^ 
cOstj-rbenefit measures will continue to be important, particularly in the 
current era of budget cqns'traints and intensifying scrutiny of gotremmentar 
activities. As sucH^, RSA can profit from use^of a cost-benefit measure in 
torms of public relations value, as viol\ as in terms of improving its own 
self^cValuatibn^dapacity. ^ 1^ * . 

Many different co^effectiveness -data elements wex'e considered in the 
design of the^ Standards*. Essentially, any data element requires in the 
denominator a measure' of program achievement and in the numeratoj: some 
measure of resources of the kind which the agency is p^rticul5iriy< anxious 
to use efficiently. We chose total^ expenditures for th6 numerator because , 
it .overcomes various accounting problems, for. example, '»w6at '^is a*. Service \ 

s . ^ > ^ ' - ^ . , . \ 

^This work Is being conducted by the T6x?s Institute for Rehabilitation 
'Research (TIRRJ , / ' 
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cost" and ''what is a counselor FTE?'* The denominators were chosen 
because of their relatively, universal acceptance as measures of ''success/* 

Benefit-cost models .^stimate total benefits and tbtal costs in terms 
of dollars. These model^ dre neutral with regard to type of delivery 
Strategy. As such they do not penalize agencies which choose to spend 
more per client in order to produce better results. Any cost-effectiveness 
measure, on, the oth^r hand, focuses on rewarding states which minimize , costs 
in achieving a given objective • Units ^of the outcome measure are assumed to 
be equal in value, i.e., one rehabilitation (26 closure) is as good as any 
other rehabilitationf. In order to offset these limitations, it is required 
that' the outcome or (benefit measure be one which decision-makers are pre- 
pared to view as having high value and units of equal value. 

The following comments illustrate the wide variety of opinions regarding 
this Standard : ' . 



• Standard "2 appears to be acceptable .as stated. 

*■ . » 

• Benefit-cost analysis is certainly one of VR's strongest selling- 
points to show a positive return of investment. However, in, 
establishing data elements and in» ultimately interpreting^ the 
results the weak points in cost-benefit analysis must be enumerated 
ptnd, if at all possible, remedied. ' 

• This Standard is the only one that deals with the efficiency concept; 
and it does it poorly by using either confusing ,and overly complex 
data elements like (iii) and (iv) or too ;broad and- general elements 
like^ (i) and (ii") . * , ^ 

4 We need information that is ustsful- at the counselor level. ' ' • 



Expei\'ditures should be broken -down by administrative, reporting, 
apJ ^specialist costs^ 

, If VR agqnci^s are^ to \>(^ evalu^ated J.n terms of cost effectiveness,' 
some attempt shoul^l be made to control for factors external tQ the 
program uhich may iihpact upon cost 'e"ff ectivenes^' related statistics. 
'Perhaps employment and other economic statistics should be. collected 
»by RS^ and reportcd'as supportive evaluation elements^ 
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• State standard in terms of j^'maximizing'^ the return to society. 

• There- is a conflict between Standard 1 and Standard 2. Do 
you try to serve a lot of people, or do you try to find jol5S 
for those who can work? 

The benefit-cost model is admittedly a complex formula. However, its 
components capture well-established and well-understood concepts, such as 
total agency costs, case service costs, and client earnings at referral and 
at closure. 

The next two comments both involve the inclusion of additional informa- 
tion in the data elements. Obtaining information useful at the counselor 
level would necessitate Adding information on e^enditures and caseload for 
individual counselors. This is. something that individual states can 
do, if they so choose, by incorpofliting counselor identification information 
into their financial and cas^oad management information systems ♦ It would 
not be appropriate, however, to. include this information in a national 
standards system. 

The suggested sepairation of agency 'expenditures into its components 
was considered and rejected for the" cost-effectiveness data elements. It 
was rejected because the dif f «^rences^ across s'^ates in accounting procedure's 
and budgeting conventions would complicate an otherwise simple and intui- 
tively obvious data eli^ment.. 

The suggestion that the Standard should be stated in terms of 
^ maximization indicates a degree of misunderstanding regarding the nature" 
•of this- Standard and its data elements. The ^'positive return to society" 
is addressed by the benefit-cOst measuf'es. Demonstrating a positive return 
to soci-ety is a minimum condition for l^iscal responsibility^ It is a con- 
^ straint, jiot a goal.- "Maximizing" the return, on' the other hand, would 
create too strong a disincentive to -serving those clients with a low 
probability of success ('e.g.\:he severely disabled). This could lead to 
over-achievement in tht? fiscal area at the cost of under'-achievement in 
the ^service delivery area. 

• ^ ■ ' • O 
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Finally, a reviewer pointed out the conflict between Standard 1 and 
Standard 2. This problem is common to most or all the Standards. The 
goals of VR are often in (Conflict, and this conflict is reflected in the 
Standards. The important thing to remember here is that the Standards must 
be viewed as a whole. The aim of an agency should not be to fulfill the 
mandate of one standard or another, but to achieve a balance among all the 
standards. 

Recommendation ' ,' - , 



Standard 2 should be retained in its present -form/ 



DATA ELEMENT 2 (i): EXPENDITURE PER COMPWITIVELY EMPLOYED CLOSURE 



/ 



.This da-ta element compares total agency expenditures to the number of 
competitively employed 26 closures. It a;^plies the most stringent criteria 
to the measureinent of cost-effectiveness by focusing on only those 26 
closures who are competitively employed. Such a priority may not in fact 
be desired, particularly given the recent emphasis on service to the 
severely disabled- However, we included this d^ta ej.ement because we feel 
that, historically and even ^today, a consensus exists that"^ompVtitive 
employment is one of the higher quality and most desirable types \f Closure 
obtainable. The form and data SQurces for this element are as follows: 



Total A'gency Expenditures 

# Competitively Employed 26 Closures 



Data Spurce 


^ Da)ta Item Pdentificatlon ^ 


RSA-2 


, Agency- ^xpeiiditures = All monies {st^^.e and 
federal) spent under control of state 
agency (110, Trust Fund, S^I,^! § E) , 


RSA-500 


Select for 26 (Part 3,N,2); Competitive' 
woYk status at closure. 
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The following issues were raised concerning' data element ?Ci)* 

• There are questions regarding the interpretation/documentation of* 
what constitutes "expenditures J* There is a ne6d for detailed 
definitions of what is meant by ''agency expenditures J* 

• Expenditures are those of an agency; similar benefits are not; 
included. Comparing states with and without similar benefits 
doesn't work* ' ~ 

• This element do^s not deal with the actual cost of 26 closures. 

' i 

it should calculate the average cost era 26 closure. 

• There is a very higli negative correlation between rehabilitationX 
rates and unemployment rates. Adjustments ^re needed for^ cost of 
living and unemploymentyfor each state, 

• Closing sheltered emplo 




creates skewed] data. It is actually 
non-competitive, but wQuld BS-^ded as competitive. 



This data element;^^ ignores non-competitive closures. It costs money 
to get them where they are. 



Total ^ency expenditures are taken from form P|SA-2, Annual Report fqr 
Vocational /Rehabilitation. They include the following: / 



1. /Obligations from Section 110 of the Rehabilitation Act, Part lit. 




>line 8: Total Expenditures, 



/•. 



/ 



Obligations £roi]i Social Security Trust Funds and SSI Progr^y 
Part IV, line 4: TcJtal Obligations and ^Mowances. / 

Obligations Trom Section 120 of the Rehabilitation *ActV Part V., 
line 7: Total Expenditures. 

* The issue of similar benefits fo^rst arose undqr data element 1 (ii). 

^ 

We repeat that while such Benefits do influence state* ]>erfofinance , taking 

tfre value dr simile 



into account' all interstate differences in 



Lar benefits 



^^/^cn computing a data element is a" Very difiAcult and complicated matter 
for' which the cost would far putweigh the increase in usefulness. The * 
Support.iyc Evaluation, however, will identify interstate differences in 
sinilar 'benefits, and these will be taken into account when recommen'ding 



correct i ve act ion . 
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The cost-effectivene'ss measure should not calculate the average cost 
of 26 closures for two reasons • First, calculating the average, cost of ' 
26 closures requires the use of life of case costs which do not account 
for inflation; also, they are Ifess' reliable ^than measures of current costs. 
Second, the ''costs" in cost-effectiveness should not be limited to expenses 
directly attributable to 26 closures. Rather, it should take into account 
all expenses incurred during the course of producing the desirable results 
(i.c.y successful rebabl^itartions) , The extent to which agencies incpr 
expenses' with und^osirable results • e . , 28's and 30's) is a measure of the 
inefficiency of that agency, 'it is necessary to include the expense's attri- 
butable to ''failures" and "partial, successes" when assessing cost effective- 
ness . ' ' ' 

Recommendation - . . 

The main weakness of this data element is that it excludes non-competitive 
closures from the compu'tatioij . This implies that a non-competitive closure 
represents "no gain" to the clidnf so closed. BPA does not wish. to imply 
this, and thus it might be reasonable to exclude the data element from the 
standards. However, this .data element ',s weakness is al.so its streng/th. 
That is, the data element allc^ws agencies to measure the cost-effectiveness 
of their services in terms of VR's highest goal achievement of competitive 
employment. Ci\rcn this, and th^ fac/^hat data element 2 (i'i) gives credit 
for non-eompetitive closures, we recommend retaining this data element. 

DATAv eiEMENT -2 - fii) : EXPENDIj'URE PER 26 CLOSURE 

This cost -effectiveness measure relaxes the measurement criteria 
somewhat to a flow '"credit" for all types ,olF rehabilitations. ^It recognizes 
that some clients are not ca-^able *of achieving competitive employment and 
that other employment outcomes can represent achievement commensurate with 
a client's abilities. This data element compares total agency expenditures 
to oJU 26 closures, ^thus capturing the effect of gainful activity, whether 
it lies jn^the realm of competitive or ^onrcompetitivo employment/ The 
form and dajta sources for this element are as follows . ^ 
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Total Agency Expendifttye 
# 26 Closures 



Data Source 


Data Item Identification 


RSA-2 


Agency Expenditure = Same as 2 (i) 


RSA-101 


26 Closures 



As was the case with the previous data element, concern was voic^^^'^^^ 
the definition of expenditures, the inclusion/exclusion of similar benefits-^ 
the desire for an ''average cost per 26*' figure, and the need for cost of 
living and unemployment rate adjustments. These are discussed in the 
previous section. Additional comments were as follows: 

• "We want comparison of similar states, not nation-wide. 

• Investigate pos-sibility for inclusion of non-26 's whose 
quality of life has improved' as a result of VR service^. 

• This is redundant with data element 2 (i) . ' 

We agr^e that the most appropriate comparisons are those between 
similar states and within a sfate over 'time; however, this does not 
negate the need for all agencies to strive for cost effectiveness in 
their delivery of services. 

The inclusion of non-26 closures is not appropi:iate. The ''effect" 
part of cost-effectiveness ref.ers to ^he goal an organization is trying 
to achieve: In the case of VR agencies, the goal is successful rehabili- 
tation of 26 closures. Other types of closures may be valid under some 
cilrcumstances, but they' are always ''second-best*' alternatives . and cannot 
be considered as the primary desired effect of VR. 

Finally, although somewhat redundant with data element 2 (i), it 
provides additional informatj.on useful in assessing performanc^. 

r 
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Recommendation 



This data element should be retained/ Use of this data element would 
be enhanced by development of a "cost of VR services'' adjustment, similar 
to the Consumer Price Index, as/~discussed under data element J (i) . 



3x, a5>~j 



DATA ELEMENTS 2 (iii) and 2 (iv) : 

(iii) Ratio of Total VR Benefits to Total VR Costs . 
(iv) Net Total Benefit from VR Services. 

These fwo data elements are very similar in concept, and they 
elicited virtually identical comments from VR agencies. Therefore, they 
will be discussed together, and the comments of reviewers will be addressed^ 
following the discussion. The forms of these data elements and the sources 
of the data are showti -below. 



Data Element 2 (iii) * 



^ BenafitS- 
Costs 



Data Sources 


Data Item Identification 


RSA-300 • 
RSA-2 

Follow-up Survey 


. Benefits' and costs are social discounted 
present values. In the benefit-cost 
ratio, the benefits are divided by costs 
(B/C); used to compare^ agencies. 



Data Element 2'(iv) 



to 



Benefits - Costs 



Data' Sources 


Data Item Identifi/?ation' .^'^ 


" R-50a 
RSA-2 
' To 11 ow- Up Survey 


Benefits and costs are social discounted 
present values. In the discounted* net 
present ,valu,e, benefits have costs sub- 
"tr^iCted (B*-C) ; used to compare changes 

over time in perfotmance level for a 
' particular ;agency. ' ' ^ * 
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Benefit-cost modeling of social service delivery systdtas enjoys 'c^r- 
rent wide acceptac^ as a measurement tool, Its|^se extends considerably 
beyond tKe VR field. The figures provided by benefit-cost analysis yield 
a single number which is an immediate indicator of program success. Be- 
cause of its surface simplicity, and because it is a popular sosphisticated 
analytic tool for evaluating. program worth, the benefit-cost^ of the VR 
system is included as a, standard. ' ' 

As a' review for \he Jlational Science , Foundation Jias noted, benefit- 
cost applications in the VR field are more extensive and have generally been 
more sophisticated (or at least at a higher level of technical quality) 
than in most other social 'service and manpower program areas ."^ There are 
a number of models available for use. In one case, RSA commissioned the 
development of a model for, routine use by the program, which was designed 
to be adaptable to the needs of many users (i.e.y state 'agencies, RSA con- 
tract^ evaluation studies, RSA itself) ancP to be capable of periodic up- 
dating and refinement as new data became available. That tlTodel, d^^eloped 
at the University of California, Berkeley and subsequently refined by BPA 
staff, has been used by RSa, several state agencies, the Urban Institute, 

Abt Associates, National Analysts, and Greenleigh Associates, among others,, 

2 

usually under RSA recommendation. This model is the basis for the two • 
data elements proposed for use in me.asuring benefits in relation to costs 
in terms of: • | 

I ^. (Benefits) ' \ . 

- - ^atio ; — ^ — r — i ^ 
C Costs ) 



a net difference (Bene/its-Costs) 



Berkowitz and Anderson, PADEC An Evaluation of an Experimental 
Rehabilitation Project , kutgers UniverS'ity; 1974. 

^Frederick C. Collignon and Richard Dodsoit. Benefit-Cost Analysis of 
Vocational Rehabilitation Seryices Provided to Individuals ?^ost Severely 
Handicapped (ISNIH) / April 1975'. 
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Both of these formulae use the discounted pres&nt value of social 
benefits and costs, and both use th^ same components to Arrive at benefits 
and ^t costs., These components, in brief, are as follows: 

' Benefits ^ 

• discounted value of paid earnings; 

• change in output of homemaker closures; 

• change in output of unpaid family workeri; 

» 

» * 

• change in "after^hours work"|(e.g., homemaking tasks per- 
formed by wage-earning rehabilitants)*; 

• fringe benefits; » 

§ change in output of families of rehabilitants* (as a result 
of rehabilitants assuming homgmaker tasks); 

• reductions in public assistance benfefits; 

• repeater costs (a "negative benefit") • 
Costs 

• total program costs during the fiscal year, minus carry- 
over cost5*and maintenance costs; 

r *( 

* • costs borne by parties other than VR; , 

• research, training, and demonstuation costs; 

• benefits foregone by clients during participation in "VR 
services (i.e., any wages and fringe benefits foregone by 
clients with earnings at referral); and 

• client-borne costs for VR services. 



A comparison of the full costs and benefits of a VR program' can be 
undertaken from several perspectives. P^erhaps the most common benefit-cost 
perspectives are the "taxpayer" perspective and the "social" perspective. 
In taxpayer BC, we compare direct administrative and service costs of the 
VR program as vvell as the costs of other government agenc^^s providing bene- 
fits and s'ervices to^the client population (SSI, SSDI, Food Stamps, Medicare, 
other employment and supportive services) with benefits suchij as taxes that 
success'fu^ rehabilitants pay from their earnings and' savings in public assis- 
tance. Social BC takes. the broadest perspective, incorporating the widest 
j*ange of costs and benefits and including on the cost side, for'example, 
costs borne by clients and, on the benefit side, client earnings as an addi- 
tion to the, GNP. V, 
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The net benefit measure (B/C) is included amon^ the standards 
data elenent-s primarily because it is th? preferred approach of ecor^cmists 
The problem with the measure is that it- is very sensitive to the sc^'ie of 
program operation: in the c^se of VR, for example, larger agendies would 
praiuce greater total net benefits than small agencies, simgly because of 
their larger caseloads. Thus the measure is inappropriate^ for comparing 
across state -agencies, although it is potentially useful for observing 
change over tine vvithin an agency. The ratio measure (B/C) overcomes, 
the problem of agency size, thus allowing comparison across agencies. As 
well B/C can b^ used to observe change over time within a jingle agency. 

The fallowing issues were raised by reviewers- with regard to data 
elements 2(iii) and 2(iv). ' 

• Probably a necessary 'evil , but not intuitively use^. 

• Question the use of benefit cost ratio. The literature is dubious 
at best about benefit cost analysis. 

• WTiat are the costs and benefits of Homemaker and Unpaid Family 
Worker? 

0 There are a^ lot of sheltered workshops which are allowed to hire 
under the minimum wage. 

• The SSA dat^ base has limitations' because it records total payroll 
contributions on a quarterly basis. Actual earnings, whether from 
full time or part-time employment, or employment during a partial 
or a full quarter are not determinable from the data base, 

• IVe should consider the social cost of the non-employment of 
workers have **lost out" in the competition with handicapped 
workers. 

1 This data element should be eliminated for the following reasons: 
' - SSA Data Link is not yet estajplished. 

Follow-up survey questions are unduly detailed for many clients 

- The mathematical formulation is too ' complicated to be meaningful 

- Cost p/ data gathering and calculations would exceed possible 

'benefit? to VR clients. . ' 

*■ 
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* * . • 

- Standard 7 (ii) -provides a similar measure. 

Benefit-cost analyses are included as data elements because we believe 
they are the most feasible method of establishing VR's "positive return to 
socie\v/' ' Obviously, theVe are differences of opinion con^ffeming how*use- 
-^ful or meaningful t^ie analyses .^re: While, we acknowledge that legitimate ^ 
differences of opinion, may exist, we maintain our belief that benefit-cost 
^analysis is a necessary and appropriate part of the Standards system. 

The costs associated with hpmemakers and unpaid' family workers are 
tne ^ame as those for any other 26 closure. The benefits of a homemaker 
ar^ determii>ed by estimating tl;ie *'worth" of homemakers in the general popu- 
lation; that is, by estimating the dollar value of the various functions 
performed by a homemaker. The worth of disabled homemakers, is assumed to be 
some proportion (less than 1) of the worth of homemakers iff general. This 
proportion is then estimated to be the same as the proportionate. worth of 
disabled workers to nofmal workers. Unpaid family wor^rs are treated 
similarly. The value of a sheltered workshop employee is his/her market 
value, i.e., , his/her wages, regardless pf whether they are above or below 

« 

' the mijiimum wage, ^ ^ . ' 

^ The value of the SSA data is a moot point at present because the • 

• purrent model uses' clients * reported Earnings in assessing the benefit of 
VR- This issue may become relevant in the future if we ever try to use SSA 
data to,jib^rain^an estimate of clients' pre-program^ earnings; howev.er, 
it is not a problem at present. 

There is a term in the model for workers who have been displaced by 
• ^ -handicapped workers^, . The term estimates the negative impact on these 

displaced workers. The term' currently has a value of zero because there is 
• no evidence of substantial impact in today *s economy. This is, of course, not 
^ relevant to BEP or' sheltered workshop ^mploy^es. • 

< Recommendation i 

, • • • • . ' 

' Despite the complexity of the benefit-cost formula and the differences 
of opinion regarding its utility, 'we believe it has an important place in 
jf tjie 'Stand'^rds system. We recommend that data elements* (iii) and (iV) both 

1 be retained. 
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FINDLNGS.FROM THE PRETEST DATA ^ 

Table 9 pres^ents our findings on Standard 2*,\ using the protest data. 

The benef-it-eost ratio and discounted net present value (DNPV-)l indicator 

ar? calculated using a 13% discount rate. The results show very •'clearly 

that B'lind agencies must not be compared to- general "or combined agencies. 

* * 

Repeal! also that the DNPV is very sensitive to agency size, and thus is 
only appropriate for use as an over-time, intrastate indicator. 




» 



Table 9 
Stilt i sties for Standard 2 



• 

Data nicment 


Hquation 


, ^ / State 


VA 

V /\ 


PA 


M 1 




OR 


MS 
RT TND 


V 

/ 

2i 


agency expenditures 


5,020.23 


3,532.60 


4,423.92 


5,437.43 


4,385.55 


14,294.65 


// competitively employed 
26 closures^ 


2ii 


agency expenditures 
26 closures-^ 


3,892.78 


2,408.64 


4,146.89 


4,^461.30 


3,941,55 


6,834,09 


2iii 


benefits 
, costs 


9.258 


• 1K836 


10.51-6 


7.057 

1 


8 . 606 


.935 


^iv . 


benefits - costs ' 
(in $ millions) 


1 284.55 


794 . 21 


470.82 


30.97 


• 190.86 


-.3 



o 
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Data Sources: 



1 RSA-2 ' , 

2 KSA Statistics Division 

3- RSA-101 
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STANDARD 3: Vlj SHALL MAXIMIZE THE NUMBER AND PROPORTION OF CLIENTS 

ACCEPTED FOR SERVICES WHO ARE SUCCESSFULLY REHABILITATED, 
SUBJECT TO THE MEETING OF OTHER STANDARDS. *' 



traditionally, success in VR has been measured by the number of "26 
closures,'' or successful rehabilitations obtained. The VR goal is to re.- 
habilitate clients, and to ignore that goal in the standards system would 
be a serious and uncomfortTable ojnission, VR does need to know how many 
individuals it successfully serves and must hav^ encouragement ,to rehabili- 
tate as many persons in need as possible. Only one data element the 
pfoportion of accepted clients closed 26 was originally proposed for^this 
standard: 



Data Element 3 (i1 



if of 25 closures 

n of 26+2B+30 closures 



X 100' 



Data Source 


Data Item Identification 


RSA-101 


26 closures during fiscal ^ear 


RSA-lOl' 


To^aL accgptea clients closed (26+28+50) 
during fiscal year 



Tjiis standard embodies the most ^asic measurement of the p?*ogram's^ \ 

success in achieving its goal of providing quality services. ^Specifically, 

if 'we accept the rehabilitation -rate (i.e . ^percent closed 26) as an adequate 

♦ 

measure of 'success, (then by inference it follows that to the extent a state 
agency has increased the proportion of clients clos^26^ its overall service 
quality lias improved as well. 

Two points need noting regarding this standard.^ F^rst/this standard 
is concerned witlx the. quality of .clients outcomes . The standard does not 
attempt to measure the actual quality of the services Ce.g., a good vs. a 
bad training program), or the appA)priateness of the service in relation to 
the clients* needs;- or the process by which services are provided. Without 
(ioubt, each of thej;e characteristics of service provision are important in 
their impact on outcomes. However, ^ince tjiere is a universaj.ly accepted 
outcome measure in the 26' *procie is measures are less important than in 

\ - ' ' - . • 
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social programs uhich cannot measure impact. Leading researchers an^ pro- 
gram officials in a national meeting on quality concurred that if such out 
come measures exist, they should be used. 

The second point concerning Standard 3 is that by focusing ^qn the pro 
portion of 26 closures, the standard fails to distinguish among different 
kinds of rehabilitation outcomes .y For example,' by looking-at *'%26** we do 
not* know the extent to which' clients are able to earn a '^living wage J' or 
the success with which clients retain the benefit^ of rehabilitation over 
time. M^owever, these and other considerations are taken into Account by 
other of the revised standards. Regardless of. the broadness of standard 
5, it is still appropriate to retain the standard and its datXefement as 
the most basic measure of quality in the rehabilitation system, \ 

' Thre following represents a synopses *of , the comments and suggesti^s 
offered concernincj Standard 5 : 

• In spite of ^ts well-documented failings, the success ratio 



is pVobably the best single measure of -program ef fectiVen^'s . 

Using the success ratio without controlling for disal^ility 
will promote mediocrity. 

Use adjustment for % severe to avoid '^creaming/' , 
% severe (STATE) ^ #26's 



% severe (NATIONAL) #26's + #28»s + #30's 

• Divide 26 closures into 2 categories: homemakerS and those' 
competitivelv'' employed. 

• Only half- the stated standard is reflected in the single ^ 
data element, which focuses on **propprtion" and ignores 
''number.** 

. Data element* 5^i) is a valid statistic on its ojvn and does not need a 
built-in adjustment for severity of disability. I-Iowever, under some circum 
.stances [eg., comparison of dissimilar ageacles), the data element may be 
.nqre* jncanuiLiful when interpreted in light of Severity. The **percent^ severe 



< Human Services Re3earch Institute and Berkeley Planning Associates, 
Assuring the Quality of Humdn Services: *A Conceptual Analysis . Berkeley, 
Calit'ornia, January 30, 19S0. * 
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data* e-lement that was recommended for inclusion under Standard 1 should 
serve tliis^purpose adequately. This is -rtot done algebraically,^"but rather 
by ^ comparing -performance on the two data elements. 

<JVe oppose the division of 26 cliJsures into homemakers. and those com- 
petitively employed. v'Tadding'* 26 closures with homemakers will show ly) 
when conffjaring t^e results of ^Standard .2, dat^ elements (if and (ii) , and - 
in Standard S, data element (i) . Thus, we believe that homemakers and 
comoetitivfely employedv^rk^rs- should bpt.h be included in this standard. 
It is true thtrt'data elemeat 5' (i) is a proportional measure of 26 
. closures; however, it does not enti^rely ignore the number of such closures. - 
In the reporting, system*, both the- numerator and denominator will be reported 
along with the ratio. The numerator is the '^number" referred to in the 
Standard. Nevertheless / it would be appropriate for this standard to include 
a data element vyhi.ch focuses only on the number of 26 's without relating 
them' to 28 's and 30 'sT To this end, we suggest including a new data element 
under 'standard 5: the net 'increase Of decrease m 26 closures in a year 
compafecl to the pri^vious year, uata for both components would come from 
the RSA-lQl. ^ • . 



FINDTMGS FROM THE PRETEST DATA - *' 

- V 

Table 10*;)Vcsents the results from the analysis of the pretest data 

pertaining to Standard 3. Data for this data element were obtained from the 

RSA-lOl. AS can readily be ascertainelSV each of the six statQ agencies 

successfully clqsed at, least 50% of their closed cases during fiscal year, 

'1979, with ^alues ranging from a low of\52,5% to a high of 85.4%. 

This Table will be more informative when it is augmented by the pro- , 
>^ ' • 

posed new data element. ^This element, a stati^stic showing the net annual 
increase* Or decrease in 26 closures, will provide a fuller indication of 
whether agencies are maximizing the number of 36 closures. 

Recommendations 

TJie comments of reviewers cinil our own analysis indicate that Standard 
3 and its data el'ement arc acceptable but' insufficient . The Standard's 
mandate for maximizing the number of 26 closures should be given equal weight 
with the mandate for maximizing the proportion of 26 closures. We tJ)ierefore 
recommend .the following: 

1U6 - 



Table 10 
Statistics for Standard 3 



Data Element 


Equation 


Stat« » 


, VA 


i'A . 


-MI ^ 


DE 


OR 


BLIND 


3i 


of 26 clojuyes " 


■ . 62.4% 


73% ^ 


— Hi 

52:5% . 

4 


t—' 

76.9% 


67.4% . 


^85; 4% 


If of :ji6+28+lQ__gJ/asur'e5 



Data Source: RSA-101 
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Both-'the Standard and the data, element sKould be retained 
in their present form. ' . , " 

A olew data element should be adde.d to this Standard. 

r. • • ' 



(n26 closijres in currenrt year) minus- 
("26 closures, in. preceding' year) 
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STXMpAROn: RElj^glLITATED CLIENTS SHALL EVIDENCE INCREASED ^ECONOMtC 
. ^ . II?DEPENDENCJE> . , " 

Data Elements: (i) * Percent 26 closures with^ weekly^ earnings at/ 
^ . , abqve federal mi'ijimum wa^e ' . 

. • ' " (ii) -fi^Coinparj^on of earnings of competitively-^employed 
• " . 'v* ' 26 closure? to earnings of employees i^iT statje * 

^ (iii) , Comparison of earnings before and after VR 

services J ~ . • 

. ' I (iv) Coip^s^son of 26 closures with public assistance 
' as priniary source o^ support beforehand after 

' VR service \ * , # • ^ 

: ., ■■ .- . 

VR^s most basic purpose is to asli'st disabled persons in finding 

^gainful employment.. The QXtent to which clients improve their ability to 

be economically self-sufficient (i.e/, "independent") through gainful ' 

■ .1 • 

'femploympjit is a fundamental concern of VR. Since most clients have low 
leve^ls of economic ^elf-sufficiency upon entry to VR, this standard is 
stated in terms of improving economic independence. 

Achievement of economic independence is one facet of closure quality 
of interest to VR, and thus this standard is included in 'th\t group of 
standards concerned , with the quality of services ; however , "economic inde- 
pendence" means different things depending on the type- of 26 closure 
obtainedv Thus, a variety of data elements are needed •to capture ^the 

concept . ^ . ' ' 

\Ti addressing the measurement of increase in economic independence for 
clients .rehabilitated, . the logical place to look is to wages and wage 
increases. Three data elements are used to assess wages: comparison' to 
the national standard- (the^minimum(^vage) ; comparison to state norms; and; 
comparison of earnings after closure to earnings prior to entry to VR% A 
vfoui^h data element considers a different aspect of .economic independence: 
that of .dependence on public' assistand'e . That, data element compares tfie 
porportion of clients at referral'and ai closure who rely on public 
a5sistancc for their primary source of support. 

V 

* ■ 7 
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The following is a summary of the .comments offered concerning the 
Stand arcT: ' * * ^ / ^ ^ 

^9 It is difficult to measur'§^ homemaker 's or^unpaid fami/fy |^ * 

• worker 's^ increase in econoiaic independence in terras 
ficiently quantifiable t«^ establish performance limits. 

^ Client income data is difficult to obtain an4 unreliable. " 

. ^ • The S^^dard addresses ^ target group that state agenciefe 
have not been instructed to emphasize (i.e. ,^ competitive 
closures) . . ' ^ , . 

• Emphasis on competitive closures and earnings leviels could*. ^ . 
' prove a disincentive to serving the severely disabled. 

Part of this Standard overlaps with Standard 5; they should ^ 
be condensed into a 'single Standard, 

It is true that measuring ^*econpmic independence*' for homemakers ^nd 
unpaid family workers is difficult, - Data element 4(iv) represents an 
attempt, to resolve the difficulty of jneasuring the degree of economic inde- 
pendence of homemackers and unpaid family workers. 

We would note several points in response to the comment regarding the 
unreliability and, difficulty of obtaining client income data. First, income 
data is not, or should not be particularly difficult to obtain. Successful 
clients are s^upposed to be functionj.ng in the closure occupation for two 
months ^prior to cl*bsure. This is adequate time to obta|.n such data^ partic- 
ularly siiicc counselors (if^they are doing their job) must contact tne client 
prior to cTosure to verify that the client has maintaine^-^^mployment for the 

4 

two month period. Second, as far as reliability goes, it is true that some 
relial)ility problems a^xist. However, in an earlier. BPA study we found that 
those consist mainly of^ "overstatement" problems (i.e., the R-300 reports 
more than. the client reported}/^ Further, the degree of discrepancy was 
smal4^ cnou);h, for the total group,* to be of little concern in relation to^ 
bur propo:^cd data elements, which dpal with rather gros:^'''^indicatoj?s (e.g., 
'^earning at least the minimum wage*'} and with agency-wide averages. Third, 
and finally, the fact that client income data may have some reliability 
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problems does not outweigh the fnaj-or relevance of client earnings to 
evaluation of state agency performance. The fact is that the comment 
regarding client income data indii'ectly criticizes the agency's own data 
base,: data elements 4(i), 4(ii), and 4(iii) all use income data obtaii^d 
from" the R-300. As we have stated over and again, we would prefei^to 
have income data based on the proposed SSA data link, but ir\ £he absence 
of an annually updated data link, and inasmucl^ as t^e^^R-SOO income data 
has been used continuously by RSA and state agencies for evaluation and 
pub^Iic relations purposes, we see no reason why, \iow, the reliability issue 
should provide 'adequate rationale for discarding the s^Candard- 

" We ^gree that this standard implicitly emphasizes competitive emp^loy- 
ment closures and- that*, on its face, the standard might discourage service 
to the severely disabled^ however, two points are rejevantf in response, 
First, we have included data elements within the total standards package 
which "give credit" for non-competitive closures, and which would be taken 
into consideration in evaluating a state 's ^overal 1 perfoirmance , Second, to 
the extent that the severely disabled are more difficult to place in 
competitive empToymenj: (and, as discussed under Standard 1, we are not 
convinced that they aVe) the disincentive to serving SDs can be overcome 
by including the proposed "percent severe" data element in the process 
of assessing state performance • 

The overlap between Standards 4 and 5 has been noted and will be 
addressed under Standard 5, 

. There is a problem with Standard 4 that has not been voiced by the 
VR agencies* The stipulation under this Standard of increased economic 
independence implies that it is the responsibility of VR to maximize the 
economic independence of individual clients. In 'our opinion, it is more 
appropriate for VR to achieve a minimum level of economic independence for 
as many clients as possible » Any financial gain beyond this minimum level 
is, 'of course, desirable, but this is not something for which VR should 
be held accountable. ' ' ; 
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Recommendation ' 

In order to reflect more accurately the true purpose of 'VR, the word 
"increased" should be deleted from this standard. 

DATA ELEMENT'.4(i) : PERCENT OF 26 CLOSURES WITH .IVEEKLY EARNINGS AT/ABOVE 

THE FEDERAL MIN INIUM WAGE 

■ « 

\Vhen attempting the measurement of economic independence for 
' fehaT)ilitated clients, ^particularly in competitive employment, the logi- 
cal place to look is to wages. The first data element for tHis standard i 
compa:j|J| the, wages of wage-earning rehabi/litants to the "standard" of the 
federal minimum wage* There are state minimum wages which may be higher 
than the federal wage, and not ali employers must pay federal minimum wages 
under all circumstaijces . The normative implications of this data element 
are that a disabled person should be expected, under equivalent circumstances, 
to makeiat least the minimum required by law for citizens of the ILS, The 
form and data ^c^urces for this data element are as follows: 



^26 closures with weekly 
earning level at/above 

federal minimum wage 

^26 closures 



X -100 



Data Source 


Data Item Identification ' 


RSA-300 

Statistical Abstract 
of U.S. 


Select for 26 (Part 3.N.2); Weekly earn- 
ings at closure; Weekly federal minimum 
wage = 35 hours X hourly minimum wage 
(BLS definition) 


RSA-300 


26 closures 
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The comments regarding this data element are summarized below. 

• Good statistic for evaluation. 

• This data element contains two ideas: ''Are 26s working at 
of above minimum \^age?"j and "Are they working /ull-time or 
part-time?" The former is more important here. The latter 
will be picked up in 4 (ii) . . 

• This assumes that weekly earnings at or above the federal 
minimum wage represent; increased economic /independence . 

• This should be based on average weekly earnings to account 
for seasonal work.^ 

• There is no standard way of reporting weekly earnings for 
homemakers, unpaid family workers , and trainees. We need 
to sp^ify a uniform co*ding. 



• . Wages of s^v'erely .disabled clients are not likely to be cqm- 
mensurate with- competitive wages. ^ 

This dg.ta .^lemejit used *the weekly minimum wa^ figure as the standard 
rather than the , hourly wage, because the former more accurately captu'i^s 
the concept of this Standard. Iftiereas hourly wage indicates a measure of 
th^ employee's worth to the employer, total (i.e. , ^we^kly, monthly^ cum- 
ulative yearly) earnings is a better indicator or*tiie%mployee' s financial 
well-being (i.e., how much money he/she makes^ and'whether that amount can 
support him/her). If an employee is able' to work only. five hours a week, 
his/her economic condition will be» Effected by this as well as by the 
hourly rate. Thus, total earnings is the more appropriate indicator of 
economic independence. ^ " 

Average weekly eai*nings probably has gr§atet vali<lity than does pre- 
vious week's earnings, because the latter does not take into account lay- 
offs and seasonal fluctuations in the economy. While this is true in 
principle, it is not clear whether it presents a problem in practice. 

Moreover, average) earnings are almost certain to be less reliable. Finally, 
taking seasonality into account requires a delay of at least a year before 
Jata coliocti%, adding to the problem of tijnely data reporting. 
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Kp.eJny earnings jfor homemakeirs , unpaid family workers, and trainees 

.ao^ by definition "zeroJ* (Excepting, of course, any earnings such 

people may get through, work outside^ their closure, occupation./ such as , 

babysitting.) This can -be entered' direct^ly onto the R-500^ or recoded 

for analvsis in those states wlli'ch enter a ^'not applicable**^ code for 

earnings at closure^ ^ • ' ^ 

Again, we question nvhet her it is true that wages of .severely cfisabled 

clients are "not likely** to be comment uraje with competitive employment • 

FurtW^r, to the extent that wages, are lower, this can be accounted for L - 
♦ ♦ 

usin^ the "percent severe" 4ata element. 

( 

Recommendation * . • * 

Given that most problems can be dealt with adequately, we recommend 
retaining^ dairu element 4(i) In' its present form, 

PAT\ ELEMENT 4fiil: COMPARISON OF EARNINGS OF COMPETITIVELY EMPLOYED 

26 CLOSURES TO EARNINGS OF, EMPLOYERS IN STATE . 

In thia data. element , the wages of rehabilitants are compared to a 
standard or w^ge fate for the general population, as' in (i). In this 
-instance, however, the focus of- the comparison is mean wage of closures with 
vv'ages to the mean wage of employees with wages in the state • 

This method controls for state-to-state variation in earniijps levels,* 
where.as using tjie fjpderal minimum wage- *5s a denominat^or does not. Other- 
wise, the concept behind this data element is the same as with (i) : ^to 
compare the wages of rehabilitated disabled clients to those of the **general 
population. In some respects, this is a more cfettiprehens i ve indicator than 
data clement (i) , because it provided an estimate of clients ' **standard of 
living" relative to other persons in the state. \ In general, as wage levels 
increase, so does the cost of living,) and the amount of income required to 
maintain an "acceptable** standard of living. Since the data element incor- 
porates cost of living (via state wage norms), \\e obtain a better measure 
of clients* living standards relative to the^surrounding environment* In 
contrast, the federal ^minimum wage is not set with reference to local cost 
of livings considerations. The form and data sources are as follows: 
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Mean weekly earnings of compe- 
titive employed 26 closures 
Mean weekly earnings of employ- 
ees y\ st^te > ' 



\ 



Data Source 


Data Item I4entificati.on ^ 


RSA-300 

✓ 


Select for/competitively employed 2^ closures ^ 
(Part .3.1.1. plus 3.1.3); Weekly eatnings at 
closure ' 


U.S» Bureau of 
Labor Statistics 


Employees = production workers in manufactur- 
ing industries 



The major criticism of this data element: focused on the appropriateness 
of the denominator. 



o The data element is a good evaluation tool. 



• . Ke like the id^a. "It is a realistic appraisal for people who 
"work in the state. ' 

/ • The data e^lement is acceptable provided RSA can get composite 
figures for the mean^wages of "emi>loyee"s in each stajLe. 

* Using jproduction workers' wages as a divisor seems to be too 
limited^ a comparison. Their inflated wages in Some states 
will bias interstate comparisons. Furthermore, 26 closures 
are -new entrants into the Tabor m^rl<;^et axid can be expected 
to earn 25-30% less than workers in general, possibly lead- , 
ing to misinterpretations of the dati element. 

Caution must be exercised when interpreting ^y single data elemf3nt, 
to ensure it is considered in context. For example, the wages of newly- 
placed cliea]:s will, of course, ^ot equal wages of production workers. 
This problem will be particularly troublesome in states with highly skilled 
or highly unionized industries; however, it is awkward and costly to com- 
pute the average wage upon entry into the labor market. This measurement 



115 



104 



provides a basis for year-to-year comparison within states to determine* 
the extent to which an agency's placements ajre keeping pace with the 
general economic conditions witliiA a state. 

s 

Recommendation 



This data element should be retained -in its present form. 



DATA ELEMENT 4(iii): COMPARISON OF EARNINGS BEFORE j|fc/b AFTER VR 

SERVICES • 

f . .The first two data elements discussed compare wages of rehabilitan^ 
with the wages of tbe general population or a .federal standard. This data 
element, however, is concerned* with the difference in the client's own 
wages before and ^af te;x the rehabilitation process, 'It is generally expected 
thftt persons in need of rehabilitation services shbuld be able ^to increase 

. their wag^s aftex rehabilitation* In a number of /cases, however, wages will 
not increase but will remain at their pre-rehabilitation level or will even 
decrease, especially in the case of a newly disabled person (e.g., spinal 
cord injury/) "^ho is to \/e retrained for a new occupation. 

T\^^a valriations of this data elementt were considered: one using weekly » 
earriitvg^data from the R-300; and one using yearly earnings data obtained 
from (the ^ocial Security Administration (.through a new 'institutional "data 
linj$y^. In the former computation, 'we would obtain the mean earnings of 26 

-■closures* at closure^ and divide by the mean earnings o£ those clients at 
referral. , Such a measure'would indicate, for the group •concerned, the 
ratio of the average closure earnings to the averag^referral earnings. 
The chief drawback to this statistic is the questionable *va^idity of , » 
weekly earnings. Also, we do not have standardized (i.e., constant) 
dollar figures for earnings at referral, since referral times vary within 
any given closure; cohort. 

In order to overcome the first methodological weakness, we strongly 
recommend efforts toward instituting an' annual SSA data lii^k. HopefyJ-ly, by 
using the data link we can overcome the validity problem (i.e./w^^ly vs, 
yearly figures), since we get t longer-term assessment of earnings potential 
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pre- and post-VR. Also, presumably, the data would bei^more reliable (i.e., 
mo're accurate) • y \ 

The^form and data sources for this data element are as follows: 



Mean annual earnings of 26 closures 

year following closure 

Mean annual earnings of 26 closures 
year preceding referral 

or r 

Mean weekly ^earnings of 26 closures 

at closure A 

Mean weekly earnings, of 26 closures 
at referral 



r 



Data Source 


Data Item Identif i\:ation 


SSA Data Link 
Not Available 


Preferred approach 


Follow-up Survey 


Select for 26; Earnings year since closure 


Revi*sed^RSA-300 


Selecy for 26 (Part 3.N.2): Earnings year 
prior to referral . 


» ^ : 

RSA-500 


Select for 26 (Part 3.N.2J: Weekly earnings 
at closure 


RSA-3(fO ' 


Select for ^6 (Part 3.N.20: Weekly e'amings 
at* referral • ^ • ' . 



Reviewers' comments regarding this data element indicate that it is 
appropriate in concept but unworkable in practice : 

• This data ^element is most appropriate given the definition 
of Standard 4^. x 

• This data element is a good measure of client* gain. It com- 
pensates for the bias in 4 (i) and* 4 (ii) toward expending 
a disproportionate share of VR resources on higher function- 
ing and higher potential clients. 



's 
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We should distinguish cases by 'training, physical . restora- 



tion, long tAm cas^s. * 
• We should^ compare^ weekly earnings pre- e 



We should^ compare weekly earnings pre- and post-VR, rather 
than annual earnings, because in some cases the latter will 
involve comparing pre-disability wages with post-closure 
wages. ' ' 4 

t This ^ta element has a built-in bias as clients ajre virtually 
unemployed at referral ,^ and -large gains in earnings at closure 
would naturally be expected. 

t The data element does not adjust' for inflation. * 

t Counselors would have difficulty collecting the required \iata. ^ 

t SSA Data Link limitations: 6SA records payroll contributions 
on a^q^uarterly basi^. Actual earnings, full-time or part- 
time status, and employment during a full or partial quarter 
could not be determined. • — V 

^ Many, of these points are valid. For example, we might expejHi training * 
cases to show a Higher earnings gain than physical restoration casg^. Also, 
the, element, as *note^ earlier, does not adjust for inflation. The compai*ison 
of weekly earnings founders on the fact that very few clients are |mployed 
at referral. To say that their earnings are higher ^t closure is almost 
trivial. At the other extreme, use of annual earnings underestimates the 
impact of VR by comparing earnings *at clost(re to earnings prior to disability 
in many cases. The true impact of VR can ie measured only through reference 
to post-disability, pre-referral earnings, which represent the *'best*^ that 
a disabled client can do on his/her own. Unfortunately, estimates of post- 
disability, , pre-referral earnings must be made on an individual basis<»-- 
an enormously complex task. While development of these estimates may be 
desirable;, it would entail considerable effort and, once implemented, would 
require complex computer programming to correctly represent each client's 
individual, circumstances. 

These methodological problems are, in our opinion, not sufficient In- 
an^d-of themselves to warrant deletion of' the data element from the standards. 
For one, this is one of the few data elements v^hich attempts to measure • 
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client change pre- versus post-VR. Such a measure is needed for a program 
which purports to improve the functioning or well-being of its clients. 
Furthermore, according to administrative staff interviewed during the pre- 
test, this data element is currently used in some states as an evaluation 
statistic (using R-300 data). Thus, the data^element appears to have 
utility for state managers, despite its methodological problems. 

Despite these positive points, we do perceive an issue with regard to 
this data element when seen in the conJ:ext of its standard. Specifically, 
we*have changed ^he focas of the standard from one of (maximal) increase 
in econoijlic independence to one of ensuring that clients achieve at least 
minimal economic independence. Whfle we still wish to maintain a concern 
for measurin^^lient change, to do so in the context of this standard would 
be inappropriate.^ For this reason we would recommend that the data element 
be removed from the standard, and that a variant of thisy data element be 
used in a new standard (discussed later) \^hich mandates that rehabilitated 
clients ^shal 1 evidence change in terms of vocational and/t)r functional 
improvement. ^ k 

Recommendation 

Remove data element 4 (iii) from Standard 4 and use a variation of the 
data element in a new standard on client change. 

DATA ELEMENT 4(iv): COMPARISON OF ^6 CLOSURES WITH PUBLIC ASSISTANCE 

AS PRIMARY SOURCE OF SUPPORT BEFORE AND AFTEK VR' 
SERVICES 



This data element addresses a question central to the meaning of gain- 
ful employment. Many VR clients will have relied c5^ public assistance for 
their primary source of support at referral. Even if a client gets a job 
at closure,' the question remains: d^Njob earnings that do not lllow'^ 
client to be free from other forms of frpancial support, such as welfare, 
really constitute gainful emplo>Tnent? ^f an individual is able, through 
rehabilitation services, to require less public assistance, this indjp^ites 
movement toward economic independence. 
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The focus of this ^particular data elemeiit is actually on a program 
level: if there are few.er clients receiving public assistance at closure 
than at referral, there is the indication that the economic independence 
of the group has been increased. 

The data element takes the following form: 



^- 26 closures 
assistance as 
of support at 



with* public 
primary /fource 
closure 



n 26 closures 
assistance as 
of support at 



with public 
primary source 
referral 



Data Source 


, . i . 

Data It^m Identification 


Revised *RSA-300 

or , • 
Closure Survey 


Select for. ;26 (Part 3.N.2.); An individual's pri- 
mary source of support is simply his/her largest j 
source, even 'it it is less than 50 o fR-300 defin- 1 
ition); Public assistance = SSA, AFDC, SSDI, or 
public insti*tution.* ♦ * 

Sel§ct for 26 


-m 

RSA-300 


Select for 26 (Part 5.N.2.) 



The comiTients below indicate the variety of reactions to this data 



element . 



• This seems like a fairly good measure when applied to 
program effectiveness with regard to disabled public 
assistance recipients. ' ^ 

• A good figure to give to legislators. 

• Obtaining timely and. accurate public assistance data is 
a problem. 

• Data are fraught with ernor. 

» The approach is inaccurate, unsound and unreliable. Drop 
it. - ' 
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• This data element tells you the least amount of information. 

• Data element must be changed. The percent of clients re- 
ceiving PA should increase by the time this cohort of 

K clients leaves VR. 

• Delay data collection to the -follow-up survey and do not 
include SSI or SSDI in public assistance. 

• -Data element (iv) should be omitted because: 

the closure survey or revised RSA-300 would not 
necessarily yield hard data on public ^ssistance; 
this could only be obtained from SSA Data Link which 
is not yet in place. 

The required use of similar benefits may result in 
higher percentage of .clients receiving public assistance 
at closure. 

This data element has an inherent problem. All the other data ele- 
ments are constructed so as to place, a positive value on things which are 
considered to be beneficial (although not always possible) for all 
clients. This is not true in the case of public assistance at closure* 
In some cases, decreased public assistance reflects the client's greater 
self-reliance and is therefore an indirect measure- of gains achieved 
through VR. In other cases, an increase in public assistance occurs when 
a client, through the intervention of a VR agency, receives that to which 
he/she has always been entitled* A decrease in public assistance would, 
in this case, be a detriment to the client and a failure on the part of 
the agency. 

Recommendation 

Given the fact that this data element cannot be interpreted consis- 
tently, we recommend that it be deleted from Standard 4. 
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FINDINGS FRaM THE PRETEST DATA " < 

« * 

Table 11 presents the results of the analysis of the pretest data 
for Standard 4, Data for this analysis were obtained from a variety of 
sources including RSA-300, th^ revised RSA-300, the U. S. Bureau of Labor 
Statistics, and tlie Statistical Abstract of U. S. The first alteimative 
measure ^or data element 4 (iii) was not included in this analysis because 
VR clients in this sample did not have both pieces of information necessary n 
to compute the measure. IVhile it is possible to do simulations using 
earnings for the year post-closure for the follow-up sample, "and using the 
year prior to referral for the A and B sample, we have chosen not to, * 
Such ^^inding might be^misleading given the different sample of clients 
for the numerator and the denominator. 

This table shows how a given state agency can appear more or less 
^'successful*' depending on the particular data element under consideration. 
This bias is especially well illustrated by flie cases pf Michigan and 
Mississippi Blind. Note that Michigan -- a high wage, industrial state -- 
registers the highest proportion of clients earning the weekly minimum 
wage amon'g the six MEU's; however, precisely betause it is a high wage state, 
it registers one of the lowest figures for mean closure wages in relation 
to mean state wages (data element 4 (ii)). In contrast, Mississippi -- a 
low wage, generally non-industrial state -- registers the lowest proportion 
of clients earning the weekly minimum wage. Yet its 26 cjlosures have ongi 
of the. highest client-state earnings ratios. These different assessments 
of state performance, generated by different data element^,, would need to 
be synthesized when making an "overalT' assessment of performance. 




Table II 
Statisti'cs for Standard A 





1 • * 






. Stiitc 








Equation 


VA 


PA 


III - 


DB 


OR 


MS 
RUND 


i 


. // 26 closures with weekly 
earnings^ at/above the 
rederal minimum wage^ 
tt 26 closures 




58% 


79% 


^ 55.1% 


72% 


34.1% 


4ii ^ 


Mean weekly earnings of 
competitively employed 26 • 

closures 
^fean weekty earnings jof 
employ&s in state 


* • 
.63 ' 


• 

.54 


.44 ■ 


.42 


.59 


.60 




^ Mean annual earnings of 26 
cl5sures year following closure 














4iii 


Me^ annual earnings of 26 , 
closures year preceding referral 














Mean weekly earnings of 
26 closures at closure • 
Mean weekly earnings of, 
26 closures at referral 


5.31 


• 

3.04 


5.76 

1 


5.84 


7.54 


2.19 


4iv 


^ % 26 closures with public 
assistance as primary source 
of support at'Clo^ire 
% 26 closures with pi^i)liGf 
assistance as primary source 
of support at referral 


.44 


.11 

■4 


i 

.36 


.38 


.20 


.93 



D^ta Sources: 1 RSArSOO 4 Ratio cannot be computed with pretest dat^a 

2 Statistical Abstract of U.S.' .5 Revised RSA-300 

3 U. S. Bureau of Labor Statistics, , i r, ^ 

Vol. 26, #5 I . . ^ • IrSq , 
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STANDARD 5: T HERE SHALL BE MAXIMUM PLACEMENT OF REHABILITATED CLIENTS 
IN TO COMPETITIVE EMPLOYMENT. NON-COMPETITIVE CLOSURES 
SRALL BE IN ACCORDANCE WITH THE ITOP feOAL AND SHALL REPRE- 
SENT AN IMPR0VB4ENT IN GAINFUL ACTIVITY FOR THE CLIENT. 



Data Elements: 



(i) Perc^t 26 closures competitively employed 

(ii) Percent 26 closures with hourly earning at/above 
federal minimum wage 

(iiiy Percent 26 closures with competitive outcome or 
with non-competitive outcome and non-competitive 
goal 

(iv) * Percent non-competitively employed 26 closures 
showing improvement in gainful activity 

Like Standard 4, this standard concerns the quality of closures 
obtained by VR agencies. Historical ly_, competitive employment has been 
seen as the best kind of closure. However, competitive employment may not 
be ;:he appropriate placement for all clients. Still, VR regulations require 
that any placement of a successfully closed client be into "gainful and 
suitable employment "consistent with his/her capacities," whether in 
competitive, sheltered, \r non-competitive employment. 

Given these requirements, we have tried to incorporate several con- 
cepts into Standard 5. Before discussing the data elements in detail, we 
present overview of tile concepts, the general focus of which 
is on gainful employment; thus, the data elements measure the following: 

1) the extent of competitive employment closures, since competi- 
tive employment 'still can be seen as the Best type of closure; 

2) the extent to w}y.ch 26 closures earn the hourly minimum wage, 
as an indicator of minimum standards for gainful employment; 

3) the extent of "goal -switching'* (i.e., going from an IWRP goal 
for competitive employment to closure in non-competitive 



^ Rehabilitation Act of 1973 , Section 302 (b) (2> (B) . 

^Federal Register, "Implementation Provisions," 1361, 1 (bb), 
November 25, 1974. 
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femployment) y with the concern that non-competitive closures 
shall not be used as a "dumping ground" for unsuccessful clients; 
and ^ ' . 

4) the extent to which non-competitive closures have obtained some 
benefit from participation in VR. 

To reiterate, this standard is concerned with the quality of closure, 
a.s evidenced by increases in gainful activity whether of a paying nature or 
not^ Competitive employment is seen as the highest form of gainful activity; 
however, allowance is made, for the need to make some non -competitive 
closures. Still, in those cases, VR is to ensure that the following have 
occurred : 

1) the non-competitive goal was planned in advance, rather 
than chosen "de facto.," and 

2) some benefits in terms of gainful activity were obtained, 
even if not of a vocational nature. , t 

The following is a summary of the comments offered concerning Standard 
5: ^ 

• Standard fs worthy, but getting reliable' d^ta for data element 
(iii) *and data element (iv) will be difficult; 

• "Improvement in gainful activity" is ba$ed on subjectiye 
It* judgment; 

. • If closure must be evaluated on caserby-case basis, how 

will their data be '*pulled together" for reporting purposes? 

• Standard reflects bias toward rehabilitation into competitive 
employment; 

• l^ile this is a worthwhile goal, it overlaps part of Standard 4 
' . and the two overlapping parts should be collapsed into a single 

Standard . 
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This Standard's bias towatd competitive employment i^'. both deliberate 
and appropriate. It refl.ects the belief that vocational rehabilitation 
should focus on employment , preferably competitive employment. At the same 
time, the Standard, acknowledges that competitive employment is not always 
possible. In the latter case, the client may reasonably expect some improve- 
ment in gainful activity as a result of^VR, That this improvement is 
measured in subjective terms poses some data problems, b^-t it does not 
disqualify such improvements from consideration. (The specific data problems 
are discussed under data element 5(iv).) 

Recommendations 

Tliere is a single recommendation for Standard 5. 

• Eliminate the words shall be in accordance with IWRP goals...** 
from the Standard. This reflects the facr that IWRP goals can, 
and at times should be, changed to reflect the progress of the re- 
habilitation effort. This will encourage clients and counselors 
alike to aspire towards a competitive outcome even when the original 
IWRP goal was non-competitive. Further, it should remove any 
incentive to ^et non-competitive goals in the original 'IWRP. 

DATA ELEMENT 5 (i) : PERCENT 26 CLOSURES COMPETITIVELY EMPLOYED 

' For a standard emphasizing maximum placement into competitive employ- 
ment^ perhaps the most obvious data element is to count how many are so 
placed. This data element is a simple, straightforward measure of degree 
of success in placing closures in competitive employment and could be 
easily implemented as the data are readily and currently available from the 
R-300, The form and data sources are as follows: 
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n competitively employed 

26 closures 

^ 26 closures ^ 



X 100 



Data 'Source 


Data Item Identification 


RSA-300 . 


Select for 26 (Part 3.N.2 0: 
Competitive work status at closure 


RSA-300 


26 closures 
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This data e^lement evoked relatively feW criticisms from the reviewers : 

• We' should compare blind clients to other blind, and general 
to general. 

• r We might want to break down competitive employment into 

fMl-time and pdrt-time enployment. 

• Coding is' the* only source jof reporting. This could be 
fraught with error. 

• Otherwise, thjere was general agreement that this is ^a 
'^good fneasure and should be kept. 

The suggestion that special consideration be given to the blind is a 
more specific version of the previously discussed suggestion to take into 
account the proportion of severe disabilities. The addition of a ^'percent 
severe'^ data element (see Standard 1) should resolve this pyoblem. Dividing 
competitive employment into full-time and part-time etnployment is compli- 
cated and unnecessary. It adds virtually no new information, and* the 
variation in hours worked is already taken into consideration* in other 

t [ . - - . . 
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data elements* Finally, while it is true that coding could be a source 
of error, it can be effectively minimized by monitoring, double-checking, 
and other forms of quality control* This should occur as a matter of course* 

Recommendations 

We recommend that this data element be retained as it is* 

DATA ELEMENT 5 (ii) PERCENT 26 CLOSURES WITH HOURLY EARNINGS AT/ABOVE 

FEDERAL MINIMUM WAGE" ~~ 

This data element applies more stringent criteria to .the measurement 

of "maximum placement of rehabilitated clients into' competitive employment*" 

It compares the number of 26 closures, with hourly earnings at or above the 

federal minimum wage to the total number of 26 closures.. As in data element 

4(i), this data element implies that a disabled person in the competitive 

labor market should be" expected to earn at least the federal minimum wage* 

Unlike 4(i) however, this measure represents an employee's worth fo th^ 

employer* Total weekly earnings are an indication of an employee '^s 

financial well-being, while his/her ''worth'* may be determined by examining 

his/her hourly wage* Thus, this data element provides a measure of the 

"value" of rehabilitated VR clients who are in the competitive labor 

market relative l:o the federal minimum wage* The form' and datk sources 

are follows: 



#26 closures with hourly earnings - 
at/above federal minimum wage ^ 
n 26 closures 



Data Source 


Data Item Identification 


Revised RSA-300 
Statistical Abstract 
of U.S. 


Select for 26 (Part 3.N*2.); Weekly 
earnings at closure divided by hours 
worked at closure 


RSA-300 


26 closures 
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The following issues were raised in regard to data element 5(ii): 

• Federal minimum wage -is a good base. 

• If a client earns less than minimum wage, it is almost 
always non-competitive , employment; therefore, 5 (ii) is 
virtually identical to 5 (i) ! 

• We believe it unfair to pay undue attention to non-competitive 
closures, since they are legal and have not been given such 
second class designation in the regulations. Therefore, we 
recommend that (ii) be deleted. * 

9 This is an important data element but, under this Standard, 
it seems to imply that competitive employment is defilied as 
employment rei][iunerated at/ above minimum wage. This is often 
n^t the case. ^ 

• Many clients are closed status 26 in sheltered workshops that 
have special certification to hire below minimum wage. 

t Comparisons of client earnings with federal minimum wage could 
result in a droi^centive with respect to serving rural popu- 
lations. This i^be cause the payoff might be perceived as being 
better for .deca eating resources to* clients in urban areas, 'where 
wage levels and therefore earnings at closure are ^much higher 
than in economically depressed rural areas. 

These comments point out the presence of some flaws in this data 
element.' In particular, the implied definition of competitive employment 
as being at/above the minimum wage causes prpblems. In some states, this 
definition is oflten inaccurate, while in other states, the fact that the 
definition is accurate^ makes this data element redundant with data 
element 5(i). The stated concern about a disincentive to serving rural 
populations, although not critical in itself, does lend weight to the argu 
ment for deleting this data element. 
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However, with some modification this data element could prove more 
useful as an evaluation statistic. Specifically, we propose that the data 
element be computed for competitively employed closures only . In this way, 
the data element, more closely addresses the concem'of the standard: it 
becomes a measure of the quality of competitive employment closures. 
Although occupations paying less than the' minimum wage are "almost always" 
non-competitive, it does not follow that all competitive employment closures 
pay the minimum wage. The task, then, becomes one of determining the pro- 
portion of competitive employment closures which fail to earn the minimum 
wage. 

Recommendation 

Modify the data element to be computed over competitive employment 
closures only. • . * 

DATA ELEMENT Sfiii); PERCENT 26 CLOSURES WITH COMPETITIVE OUTCOME OR 
~ WITH NQN-COMPETITIVE GOAL AND NON-COMPETITIVE 

OUTCOME 

i Competitive employment may not be the appropriate placement for^all 
clients. Nevertheless, VR regulations require that all placements be into 
"gainful activity" and that placements be consistent with the clients' 
"capacities and abilities," whether in competitive, sheltered, or non- 
competitive* employment. 

There is much speculation in the field over the abuse of 'Tiomemaker" 
and "unpaid family worker" .categor^ies, specifically regarding the ^use of 
these categories to ensure success rather than because the placement is 
appropriate. While, maximizing the proportion df successful closures (as 
in 3 (i)) is*" important to the purpose of VR, it does not ensure that non- 
competitive placements are suitable for the client. This data element- 
addresses the concern that non-competitive closure categories not be used 
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to salvage '^successes'* for clients who were unsuccessful in their planned 
competitive goals. The form and data sources are as follows 



# of 26 closures who havp: 

(a) competitive goal and com- 
petitive outcome ^ 

(b) non-competitive goal but 
cqmpetitive outcome 

(c) non-competitive goal and non- 
competitive outcome 

that is, excluding 

(d) competitive goal but ijon- 
competitive outcomg 

* 26 closures ^ 



Data Source 


Data Item Identification 


Revised i^SA-300 


Select for 26 (Part 3.N.20; Work status of 
original IWRP goal; Competitive work status 
at closure; Non-competitive work status at 
closure = sheltered workshop' employees, 
BEP "employees, homemakers,. unpaid family 
workers 

# 


Revised RSA-300 


26 closures 



The following is a- summary of the comments offered concerning data 
element (lii) : 

• Vhis seems like a good measure of program effectiveness, 

• Appears difficult to calculate. Will the information obtained 
be that helpful? • , 

• It overlooks the fact that vocational goals frequently ^change 
as 'the WRP progresses. Counselors vould have an additional 
form to complete. 

• Counselors say they often put the goal just out of reach, then 
change it after seeing what happens in training. 
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• Which IWRP goal is intended? The original or the final amendment? 

• Itfhy not comp^e work status at referral* to work status at closure? 

• Data element has n'o statistical value since all states are over 
90%. " . 

• Not really useful for getting money out of foundation, 

•< 

• Better left ^o aprocess/case review standard. Delete. ^ 

The major problem with this- data element involves the possible mis- 
application of the IWRP process. The IWRP is intended to be a statement • 
of a realistically attainable goal which, if necessary, can be modified for . 
a variety of valid reasons as the client progresses. through the VR process. 
That is, the IWRP serves as a guideline rather than as a hard and fast rule. 

Datar element 5(iii) reflects a concern that the counselor and client 
make a conscientious effort to adhere to the guidelines of the IWRP. ^Un- 
fortunately, that is not the effect of the data element. The inclusion 
of this statistic in the Standards overemphasizes the importance of matching 
the outcome to the goal. As such, it is a disincentive to setting ambitious 

(i.e., competitive employment) goals in the Sriginal IWRP,> and it also reduces 
the flexibility of the counselor in refining the go^^l^ durfti^.rehabilitation. 

Recommendations - 

; * ir ^ 

This data element reflects a legitimate concern about the VR process 
(i.e., the concern for quality planning and client assessment), but it is 
not useful as an evaluation statistic. Consequently, we recommend that it 
be deleted from the Performance Standards and that the underlying concept 
be addressed through a Procedural Standard. 



DATA ELEMENT 5 (iv) : PERCENT NON-COMPETITIVEIY EMPLOYED 26 CLOSURES SHOWING 
V IMPROVEMENT. IN .GAINFUL ACTIVITY. 

As stated earlier, closures * into non-competitive employment may be 
legitimate for certain clients. Nonetheless, if VR is to claim any credit 
for "rehabilitating" clients into non-competitive employment, then there 
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must be some indication that VR helped improve those clients* capacity Tfor 
gainful activity. Tf the client obtained no benefits whatsoever from VR, 
then VR^has essentially wasted money and time. Obviously, such outcomes 
are not desirable*. 

This data element takes a subjective approach to the problem of 
assessing the legitimacy and appropriateness of non-competitive closures. 
It is computed by taking the percent of non-competitively employed 26-closures- 
who^ state they have done any of the following: improved their self-care 
abiliti)^s and thus freed other family members to join the labor .force; ex- 
perienced improvement in any self-care or homemaker-related function^; 
experienced improvement in job-related skills; or had "improvements** in 
attitude.. This wide range of indicators assesses the extent to which non- 
competitive closures benefit as a result of int^ention. The fcfrm and 
and data sources for this data element areas follovs: 



non-competitiveiy employed 
26 closures showing improve- 
ment in gainful activity ^ 
# non-competitively employed 
26 closures surveyed at 
cl(^ure 



Data Source 


Data Item Identification 


Closure SufVey 


Non-competitive, work status ^t closure = 
same as SCiii); Gainful activity = release 
of other family members for labor force, 
attitude changes, aspects of self-care, 
homemaking skills, job-related skills 


Closure Survey 


Select for non-competitively employed 
26 closures, defined as for 5(iii) 
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The criticisms of this data element generally focused on the validity 
of the data: 

• We like the fact that non-competitive employment was addressed,^ 
but measurement right now is poor* 

t This data element has all the problems associated with self- 
reported information. , ^ 

• How do you determine improvement in gainful activity? There are 
numerous questions on thev survey, and only* one improvement is 
needed for the client to be included in the numerator. 

• This ought to be thrown out because it's difficult to define 
and measure in a way that allows for meaningful interstate 
comparison. 

• Substitute Walker Associates Degree of Improvement on Independent 
Living Scales. 

We feel that the concept embodied in this standard is very important, 

in that it allows assessments of achievement of gainful activity for non- 

/ 

competitive closures. However, rather than attempt to impose the original 
instrument on atates at this time, BPA recommends substituting those 
elements of the Functional Assessment Inventory (FAI) and th^ Life Status 
Indicators (LSI) which will soon be pretested for inclusion in the MIS. 
The FAI and LSI have been identified by RSA's MIS Work Group as having 

potential for providing valid and reliable measures of independent living 

r 

Status and functional abilities. The exact data elements, however, cannot 
be specified until afteif the FAI/LSI pretest. Until that time, performance 
on this data element siiould not be reported. 

Recommendation 

— 4 

Modify the data collection methodology to use the FAI and LSI compon- 
ents which are ultimately chosen for inclusion in the MIS. 
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FINDIXGS FROM THE PRETEST DATA 

Table 12 represents the pretest findings for Standard S. The cjata 
actually provides a furthei; rationale for removing data' element 5 (iii) 
from the Performance Standards. Specifically, no st^te achieved lower 
than a 90% '^success*' rate on the data element. Because a 90% success 
rate^ would be an acceptable minimum cut-off point for performance on this 
data element, there is no need for a specific performance standard to 
measure an aspect of program functioning which all states . consistently 
achieve. For this reason, we have recommended that, this element be addedy 
to the Procedural Standards which will be systematically reviewed less 
frequently than the Performance Standards. 
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Statistics for -Standard. 5 
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Data Element 


« 

Equation • 

— ^ ' ^ ' > 


State .... . ' - 


VA. 


PA 


MI 


DE 


OR • • 


BLIND f, 


5i 


# competitively employed 26 
closures! 


74% 


73.3% 


94 . 9% 


'«»'8S.5%; 


82.K 


42 ."2"^ 


#26 closures! 


5ii 


//26 closures with hourly 
ei^rnings^. at/above federal 
minimum wage-^ 


561. 


52% 


91% 


53.9% 


-7.i% 


i 

f 

, -52.2^" 

-f 


#26 closurfes^ 


Siii 


2 ' ' 
#26 closures with : ^ 

^ competitive go^l and com- 

^"'•^pctitive outcome 

b) non-competitiv£ goal but 
competitive outcome 

c) non-competitive goal, and 
non-competitive outcome 


90.5 


97% 


99.6% 


93 . 9% 


.92.7% .' 

y • h-r 


* 

r 

93.1% . 


#26 closurps^ 


5iv 


# non-competitively employed 
26 closures showing improve*- 
' ment in gainful activit?!^ 


'87.8% 
(N=49) 7 

/ 


45.0% 
(N=80) 


l 

64 . 3% 
(N=14) ^ 


\ 84.6% 
^ (N=13) ■ 


■93.5% : 
•'(•N=31) 

— ' ^ 


,9r.2%- 

(N=34) • 


# non-competitively employed 
26 closures surveyed 
at closure^ - 
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Data Sources: 1 

V 

1 

3 



^ 



RSA-300 

Revised RSA-300 
Closure Survey 
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STANDARD 6: VOCATIONAL GAINS SHALL BE ATTRIBUTABLE TO VR SERVICES, 



Data Element : Comparison of earnings change from referral to closure 
of 26 closures to earnings change of a control group 



The intent of this standard is to provide a measure of VR's effective- 
ness in its attempt to rehabilitate clients; that is, to measure the extent 
of a causal relationship between "the VR services provided and the outcome , 
Or vocational gain. Research has indicated that some vocational gain may 
occur independent of the provision of VR services. Studies suggest that 
both clients of VR and non-clients will show, over the same time period, 
increased earnings, increased levels of skill, and other vocational gains. 
This standard is justified in order to demonstrate whether vocational gains 
can be directly and accurately attributed to the provision of VR services. 

Given^the intent of the standard, we feei that it fits in well with 
two of the three basic RSA goals. The relation between the standard and 
the goals is derived from the need to establish a prior condition . Clients 
may or may not experience vocational gain after participating in VR ser- 
vices. But even if they do, the question remains: was VR in any wa/ re- 
sponsible for that gain? If the activities of VR on behalf of the client 
had little or nothing to do with the gain, then the quality of the service 
(in te-rms of planning, prescription of appropriate seirvices, and effective- 
ness of service provision) should be called into question. This standard 
is similarly tied to the cost effectiveness goal: .if VJR is responsible, 
then it has been (and can claim it is) cost-effective. Otherwise, VR has 
wasted money and effort, regardless of the outcome actually achieved by 
the client . 
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Pat a Element 6 ii) 



b. 



c. 



(Average change in earnings for 26 closures) 
(average change in earnings for 28 + 50 closures) 
Average change- in earnings for 26 closures ♦ 



(Average change in earnings -for 26 closures) 
(average change in earnings for 28 closures) 
Average change in^ earnings for 26 closures 



.(Average post-closure earnings for 26 closures) - 
(average pre-referral earnings for 10s, accepted 

clients) 

Average post-closure earnings for 26 closures 



i Method 


f 

Data Source 


Data Item Identification 


1 a. - 


or 


SSA Data Link 
(Not Available) 


Select for 26, 28, and 30; Preferred approach 




Follow-up Survey 


Select for 26, 28, and 50; Earnings year 
• following closure 


i 


or 


Revised RSA-500 


Select for 26, 28 > and 30; Earnings year 
prior to referral 

1 




RSA-500 


■t ^ — — ^ . . 

1 

Select for 26, 28, and 501 Earnings at 
closure; Earnings at referral 


h. 


or 


SSA Data Link 
(Not Available) 


Select for 26 and .28; Frefe^gJ^proach 




Follow-up Survey 


Select for 26 and 28; Earnings year following 
closure , 




or j 

i 
j 

h 


Revised RSA-500 


Select for 26 and 28: Earnings /ear prior 
• to referral 




RSA-500 


Select for 26 and 28; Earnings at closure; 
n-irnir^i^.n at referr^j] 


c* 


1 

or r 


SSA Data Link 
(}:ot Available) 


Select for 26 rir.v! iO 




Follow-up survey 


Select for 26; Earnings year following 
closure 






Revised R-300 


Select for 10; Earnings year prior to 
^ referral 
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Given the nature of this standard, a control group comparison is 
required. The difficulty involves the selection of a suitable quasi- 
control group given the infeasibility of the random assignment of clients 
to services. Three variations were proposed. Two of the data elements 
propose using accepted, but unsuccessful clients as the quasi-control group 
Tliey use wage changes of unsuccessful closures as a baseline to ascertain- 
ing the extent of '''unique'' wage gain for 26 closures (i.e,, the proportion 
of wage gain attributable to" successful completion of VR services). The 
third data element uses the clients entering status 10 during the current 
reporting year as the quasi-control group. Their pre-referral earnings 
are compared -tja-the post-closure earnings of 26 closures. 

The criticisms of reviewers focused on the data element rather than 
the Standard: 

ji This data element concentrates on financial gain and ig- 

'J 

nores other aspects of vocational gain. Increased earn- 
^ ings ia. only one aspect of vocational gain. It is often 

not the most important^ 

• Aside from the fact that ''earnings change'* is a poor measure 
to begin with, one can also question whether 28 and 30 clo- 
sures comprise an appropri^ate comparison group. In some ^ 
states, 28 's and 30 's by definition do not have earnings at 
closure (i.e,, anyone with earnings is closed 26,) In 
others, counselors don't know earnings at closure for 28 's 

and 30 's because these clients simply don't show up (l-c,, they 

are "lost*'). 

Y ^ 

• Ive recommend that the third measure (comparing 26's,to 10 's) 
be used. It should yield a more reliable index of this 
Standard, 

• The concept is good, but the data element is not. Perhaps 
the sought-for cauSal influences can only be established 
through controlled studies, 

• Drop this Standard and obtain information with a one-time 
research project. 
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Two major issues confront "this data element. First, ^ wage gains have 
been prqpased-'-asr^^a-TneasarenTf'irocatlonal'-gafins. Wages reflect neither 
hours worked nor hourly wage; they are simply aggregate earnings received 
while employed. For clients with high pay levels but lower hours worked, 
. this measure would create. a downward Mas . In addition, measuring only 
wage gains' ignores other vocational ^gains. 

Second, as control groups, 28's and 30's are not totally comparable 
to 26*s. The 28's recei^/e some services but fail to ^ajj^eve rehabilitation. 
^ They [Presumably axe thus more difficult cases to rehabilitate than 26 's, 
and the difference between their post-closure wages and those of 26 's 
inay lead to overestimating benefits of 26's attributable to VR services. 
(In short, the 28 'sj would be likely to fare worse than a true control group 
of 26 *s without services.) ^ 

The 50 's involve a client population which often drops out because of 
client lack of interest in VR program participation. Manpower studies have 
shown that early drop-outs are not necessarily poorly motivated,* but rather 
are often highly motivated clients who perceive they can make it on their 
own and don't want to waste time in manpower training. Thus, 30 's may 
be a client population with better prospects than a true control group of 
26 's without services, and the difference between their post-closure wages 
and those of 2b's may underestimate the true benefits of VR services. ^ 

A control group of a client population accepted for services (Status 
10) the same year 26 closures are tenninating VR services has a number of 
advantages. There are far more clients in Status 10 than there are in i 
Statuses 2a and 50. , Also, comparing similar years' earnings between the 
two groups eliminates the need for inflation and labor market adjustments. 
Therea^ problems, however, witJi comparing 26's to lO's. In general, the 
'lattcf\group have been disabled more recently than the former. They have 
had less time, to adjust to a critical change in their liyes. A second prob- • 
lem is^ that Status iO includes clients who" eventually may be closed in any 
one of the throe statuses (26, 28 or 50.) Thus, even Status 10 clients do 
not (constitute a suitabie" comparison group. 
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Sununary and Recommendations ^ 

TtfKiI?**tTie' concept embcT^ied in the standard is important and app:fopri- 
ate as an evaluation concern, measurement of the concept is very difficult 
Ideally, we would want to establish objectively, on an individual client 
basis., that VR program intervention was responsible for the gains made by 
clients. Unfor^nately , trie inadequacy of the comparison groups as con- 
trols prevents us from making conclusive statements concerning causality. 
These problems render the data element, useless both for evaluation and 
for management. 

Despite the critical flaws in the data element, the Standard itself 
reflects a principle important for any evaluation of VR. Consequeritly , we 
recommend the following: 

• This data element should be eliminated. 

• This Standard should be removed from the performance stan- 
dards, and the mandate for establishing causality should be 
fulfilled through periodic controlled research studies of 
clients on a national level. 

With this change, we eliminate from the standards an unsuccessful 
attempt 'to attribute vocational gains to VR services » We suggest that 
such causal analysis is .better done in special studies. However, we still 
\lf^sh to measure vocational and other gains of clients made during VR 
services, to allow description of *ha;Upositi"Ve change which has occurred. 
Given this, we recommend that the "causality'' standard be replaced by a 
•'client change'' standard, with appropriate associated data elements. Thus 
the new Standard 6 will read as follows: 
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STANDARD 6: REHAB.rilTATED CLIENTS SHALL EVIDENCE VOCATIONAL GAINS 

Ultimately, the data elements for this stSidard' shouid measure both^ 
vocational and non-vocational changes which might acoine to .rehabilitated 
clients.* However, in the short-term we restrict ourselves to vocational 
measures; in particular, to earnings measures. We recommend using the 
average pre-post earnings change for rehabilitated clients . This is 
equivalent to an equation of the form: 



Data Element 6 (i) (new): 

(Sum of closure earnings) fSum of referral earnings 

. Cfor 26 closures J " jfor 26 closures , 

n26 closures 



The data sources would be as follows: 



Sum of closure earnings 


Item 3J 
(Sum for all 26 closures) 


Sum of referral earnings 


h 

R-300, Item 2P 

(Sum for all 26 closures) 


^26 closures 
• 


R-3p0, Item 3N2 
(Tdtal number) 



^ JVe recomqi.end using R-300' data to compute this data element. We dis- 
cussed the problems with R-300 data (primarily, the validity of weekly 
earnings figures) 'under data' element 4 (iii). Despiire these problems, 
R-300 data rcjjmins the most^ attractive alternative until an annual SSA 
data link is' 6,^tablished* ^ 

This data element is very similar to data element 4 (iii),^vhich we 
have recommended for deletion from Standard 4. The major distinctions 



See pag.es 106 and 107. 
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between the two are that ^he new data element 6 (i) provides an average 
of individual, raw dollar earnings changes , while 4 (iii) provides a 
ratio Expressed in terms of a percentage change . 

We do not at this time have any recommendations for specific data 
elements to measure non-vocational change. As with, the data elements 
associated, with/non-competitive employment closures (as in data element 
5 (iv)X,the methodology for assessing non-vocational change needs develop- 
ment. This development should occur as an outgrowth of the FAI/LSI pre- 
test for the MIS. Until. such time as the measures can be developed, no' 
data collection or reporting will be conducted for this data element. 

V 

FINDINGS FROM THE PRETEST DATA . 

Table 13 presents the results of the analysis of pretest data for 
Standard 6. Data element 6(i) attempts to represent a causal relationship 
between VR service delivery and client outcome by comparing the net earn- 
ings gain of 26 closures to that of 28 and 30 closures, and to 28 's alone. 
In both cases, the interpretation of the statistic is the proportio.n of 
26 closures' earnings gaintthat can be attributed to VR services. How- 
ever, a glance at Table 13 reveijals that, for both measurements, mpst states 
achieved a proportion in excess of JL.OO (due to negative earnings change 
for unsuccessful closures). This, of course, makes no sense conceptually, 
and it constitutes an additional argument for eliminating data element 6(1), 
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Table 13 
Statistics for Standard 6 



Data Element 


j , Equation 


State- 


VA 


. PA 


MI 


• DE 


OR ■ 


■ Ms 

BLIND 


/ 


for 26 closures) - (average 
/change in earnin'gs for 28 § 
30 closures 


1.10 


.1.13 


. Jl.95 


1.00 


1.02 ■ 


1.20, 


(average change, in earnings 
for 26 closures) 


(average change in earnings 
for 26 closures) - (average 
change in earnings for 28- 
closures) 


1.10 


1.13 


1.00 


1.05 


1.02 


, 1.32 


(average change in earnings 
for 26 closures) 




Data Source: 



RSA-300 

Closure Survey 



/ 
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STANDARD 7: REHABILITATED CLIENTS SHALL RETAIN THE BENEFITS OF VR • < 
SERVICES 

Data Elements ; (i) Percent 26 closures retaining earnings at follow- 
• • , up . . ' 

(ii) Comparison Qf 26 closures with public assistance 
as primary source of support at closure and'at 
follow-up . 

(iii) Percent non-cpmpetitively employed 26 .closures 
* . retaining closure skills at follow-up 

Retention of benefits gained through vocational rehabilitation ser- 
vices is important to the rehabilitated client and as a measure of overall 
program effectiveness. Job losses following successful closure imply 
program failure and point to incongruence* of program goals vis^a-vis in- 
dividual client goals. Are we "rehabilitating" clients temporarily to 
meet program objectives, then finding clients. back where they started a 
few months later? This question has a great degree of importance to the , 
overall VR mission and thus a standard in this area is highly appropriate. 
Aside from employment measures x)f benefit retention > additional attention 
is giveri to expanding the data elements fox this standard to include non- 
employment .measures.. 

ThisjStandard embodies another of the concepts related to quality ser 
vice in the VR program. In the ideal setting, successfully closed clients 
would be permanently rehabilitated. The theory, of course, is that if 
quality services are provided, clients will'retain the ability to. function 
in a job and to compete in the labor market. The manifestations of those 
abilities are that clients do ii|:fact retain their jobs, or some job. Of 
course, it is not always possible to retain jobs (or earnings levels), 
regardless of .the quality of services provided by VR. Clients may suffer 
from unanticipated relapses or complications of their disabling handicaps, 
\yhichvcan cause clients to lose their employment capabilities temporarily 
or permanently. Alternatively, clients can be laid off from^jobs due to 
macro-economic^ conditions, regardless ^of the quality of VR services. 
Nonetheless, in general we would expect clients to retain benefits. The 
standard is appropriate for inclusion in the overall .standards system, 

• ^ 
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The following is a siunmary of comments offei^ed concerning Standard 7-. 

> * 

• While we Certainly support the idea of measuring retention- of- 
benefits, we question the implication that "recidivism", necessarily 
implies "program failure,*' The whole notion of "post -employment 
services" is based' on the fact that certain individuals need and 
are entitled to repeated contact^ with the VR agency over. time, 

• Retention of benefits is important for evaluation /'but it must 
be tied into a study of why retention occurs and what benefits, 

^ ,aje''r Gained . 

• It is important that benefits gained through VR services be 
demonstrated before their retention is measured. 

Standard 7 does not measure recidivism. It measures instead the • ' 
retention of the skills attained at closure. A client who has retained 
those skills. (e,g., a job) while maintaining contact with the VR agency 
(e.g., for post-employment services) is not counted as a program failure* 
A 'failure occurs -when a clieyy loses paxt or all of the benefits and 
skills acquired during rehabilitation, regardless of any post-closure 
contact with the agency. Further, the fact of provision or non-provision 
of post-employment services will be taken' into consideration during the 
supportive evaluation. 

Recommendation • . , . 

^ ' 

• Standard -7 should be retained in ^its present form. 

DATA ELEMENT 7 (i):' PERCENT 26 CLOSURES RETAINING EARNINGS AT FOLLOW-UP 

' Since, the achievement of "gainful activity" is the basic goal for 
the VR Client, a simple measure of the retention of that benefit is the 
client's continued employment. However, this data element tightens the 
criterion to consider retention of the clientv^s economic welfare level 
at closure. , The data for such a measure would be available .through follow- 
up int^uiry♦ The denominator includes only those clients working at 
closure, on the theory that these are the clients who had -^TDenefits" from 
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VR that need to, be "retained." Unfortunately, the element excludes those 

clients whose employment capabilities wer6 enhanced by participation in VR 

and who may obtain Jobs- at some point after closure thanks to VR's . 

intervention but who had no earnings at closure,, 

' Earnings at closure and post-closure could be compared in several 

ways, including changes in earnings summed for all ^working, clients between 

the two j)eriods, or an average" percentage change for individual clients 

across the two periods. The data element may need adjustment for differ- 

♦ 

ences between- successive periods to account for inflation. However, a 
simple "increase-decrease-stable" measure of individual clients (aggregated) 
avoids some these problems. Ideall^, data for the comparisons could 
rely on the SSA data link to enhance data quality. The form of data element 
7.(i) and the sources of data are shown below: x 



# 26 closures with earilings 
at closure who retained or 
increased earnings at 
follow-up (1 year) ^_ 

# 26 closures with earnings 
at closure examined at ,* 
follow-up 



X 100 



Data source 


Data^ Item Jdentif ication 


SSA Data Link > v 
(Not Available) 


Preferred approach 


Follow-up. Survey 
>Revised RSA-5"00 


Select for 26 with jobs at closure; 
earnings at closure and fbllow-up 

Select for 26 with jobs at c4osure, 
using identifying data on surv.ey 
instrument 
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The following is\a summary of coirtments and. suggestions for this" data 
element: ' ^ * > ' 

• The first two data elements under this standard address areas 
- relevant to an evaluation of program effectiveness. Good way 

to measure retention of benefits, 

• Sample of foll.ow-up returns may be biased, since the more 
successful rehabilitants are, perhaps, more likely to be 
contacted and return the questionnaire, 

• We recommend retention of 7 (i) , but only if the data can be 
obtained from th6 SSA data link. . 

• The SSA data base ha? limitations because it recoi^ds total 
payroll contributions on a quarterly basis. Actual earnings, 
and whether from fuli-tim^ or partr-time employment, or employ- 
ment during a partial or a full quarter, are not determinable- 
from the data bas^e. 

• Change concept behind data element to "retaining, maintaining, 
or working toward economic independence" to take into account 
job training, continuing education, etc, 

• Why are we looking at number of clients instead of looking at 
overall earnings? 

• Should vCorrect for inflation. , 

, * It is probably true that successful clients will be over-represented 
in a sample based on follow-up surveys. The data element will tend to 

^overestimate, retention of benefits., However, this^ is not a serious 
problem, for two reasons. First, sijiCe the'problem is .common to all 
survey research, the bias wil'l be consistent across states and over time. 
Thus, cross-state and year-to-y^ar comparisons will not be significantly 
affected. Second, consistent overestimates of retention can be taken into 

^account when seating performance levels. . ^ ' ' - 

The major advantage of the SSA data \ink is th"at4 it' obviates the > , 
problem of sample bias; however, , as we have 'said, this prbb^em is not 
serious. Some disadvantages of the data link were mentioned in tfee .comments 
but the main drawback, is that the , data link is Viot^ a^vailfible at present, 

•■- \- 
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tt would be inappropriate tp change this data element's underlying 
concept to "working toward economic independence.** There are other data 
elements (5(iv), and 7(iii)) that provide for closed clients who 
are not competitively employed. Data element J(i) is, and should B^, 
concerned with retention of benefits from said employment. 

This data element is' based on th^ number of clients rather than .total 
earnings- because^^the^ purpose of VR is not to maximize aggregate client 
earnings, but ,to help each client achieVe economic independence. Using 
the mean earnings ;or overall earnings would allow a- few very successful or*^ 
very unsuccessful clients to piill the statistic upward or downward. Even 
if these effects were to cancel each other in practice (which they may not)* 
a statistic based on a group's total earnings is not in principle a valid 
reflection of the Intent of Standard 7.. , 



. Recommendation f • ^ ' 

Data elj^ment 7(i) should be retained in its present form. 



DATA ELEMENT 7 fii): COMPARISON OF 26 CLOSURES WITH PUBLIC ASSISTANCE AS 

PRIMARY SOURCE OF SUPPORT AT CLOSURE AND AT FOLLOW-Ug 

This data element would pro v ide^a needed dimension in assessing bene- 
fit retention for non-competitively as .well as competitively p|p.ced success- 
ful closures » Here, benefits are proxied by \^ieasurii^g jt:he e^el^ t of the 
clients* use of public resources. By focusing onthe degree to which there 
is a reduced need for public assistance, an emphasis is given to the econ- 
omic self-sufficiency ofc the client in terms of s'tability or improvement. 
IVhUe this represents only one dimension of the possible benefits associated 
with successful closure (Jind one less sensitive to th^ complete rang? of 
effects of VR services), it has a high. degree of face validity as a measure 
of public resource burden. The form of data element 7(ii) is presented 
below; along with the sources of data. 
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% 26 closures with public assistance 
as primary source of support at 

follow-up (12 months) 

% 26 closures with public assistance 
as primary source of support at 
closure 



) - 


■J* - 














Data Source 


i 

Data Item Identification 








Follow-up Survey- 


Select for 26; Public assistance = 
same as 4 (iv) ; Primary source of 
support = same as 4 (iv) 








Revised RSA-'sOO 


Select for 26; Public assistance = 
same as 4 (iv) ; Primary source of 
support = same as 4,(iv) 



The comments of reviewers did not express any disagreement with the 
concept of this data element. However, there was some con'cem about the 
nature of the data itself. 

• Data element is satisfactory and is relevant to an evaluation 
of program effectiveness. 

' • Data element should be dropped, because' it requires a special 
' follow-up after closure* 

• Follow-up survey is not likely to yield reliable data* 

• There are problems collecting "primary source of support" at 
closure. ' - 

• Definition of public assistance needs to be tightened; exclude 
SSI and SSDI from "public assistance." 

The fact that this data element requires, a follow-u^ survey is not in 
itself a convincing argument for dropping the element. The appropriate 
criteria are the quantity and quality of the data, and the cost of con- 
ducting the survey. The instrument can be revised to increase the 
response rate and data reliability, while providing at least a partial 
reduction in processing costs. i 
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•The major problem with data element 7(ii) is the definition ^nd 
identification of the primary source of support. The '^primary source'^ is 
currently defined as the largest source, even if it is less than 50%. More- 
over, ac cord ing to the Rehabilitation Services Manual ^ ^^In determining the 

largest single source, combinations of public assistance payments should 
be considered as one single source in making the determination*'. Other 
sources are to be considered individually, and they include several private 
sources: the individual's earnings, dividends, etc.; family and friends; 
private relief agency; and private insurance. ' 

The foregoing definition can lead to some anomalies. For example, if 
a client derives 25% of his income from combinations of public assistance, 
and if the remaining 75% were divided evenly among the four private sources 
mentioned* above, then public assistance would be the single largest (i.e., 
primary) source of support. However, if another client derives 45%. of his 
support from public assistance and 55% from wages, hi.s prijnary source would 
not be public assistance. This despite the fact that, in prdportional terms 
his public support is almost double that of the first client. 

From this example we can see that 'if a client's primary source of 
support is coded as public assistance, it does no^ really tell us much 
about the extent of the client's dependence upon the government. A more 
consistent definition would involve determining the client's total dollar 
amount of public assistance and dividing this by his total 'dollar income 
from all sources (public and private). If the ratio of public assistance 
to total income is greater than ,50, then public assistance is the primary 
source. 

The method of identifying primary source also presents problems. The 
current method of making that determination using the counselor's deci- 
sion at closure and the client's decision at follow-up is inherently 
inconsistent and unreliable. A better method would be to have the client 
report at closure and at follow-up his total income and his total public 
assistance. The counselor (or the state) could then compute the ratio of 
public assistance to total income. This method will require a ^ght ' ^ 
modification (actually a simplification) of the follow-up instrument. 
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Recommendation 

Data element 7(ii) should be retained. The definition of primary 
source of support and the method for making that determination should be 
arttered—irr^e m anne r desc r ib e d abov e. — - ^ — 



DATA ELEMENT 7 Ciii): PERCENT NON-COMPETITIVELY EMPLOYED 26 CLOSURES RETAIN- 
ING CLOSURE SKILLS AT FOLLOW-UP 



This data element extends the concept of retention of benefits to 
non-competitive closures, in terms of the benefits assessed in Standard 5, 
data element (iv). The data element computes the percentage of all non- 
competitively employed clients who have maintained or improved their 
closure skills at follow-up. The -percentage is computed in terms of the 
number of clients who stated they had improved on any aspect of self- 
care, attitudes, homemaker skills, job-related skills, or had ''improved 
so as to release other family members to join the labor force." Like 
Standard 5, data element (iv) , the exact methodology for deriving this 
data element's needed information in a consistent and reliable fashion 
is problematic. The state of the art is in exploratory stages and the 
special follow-up data collection required involves significant resources. 
While the data element is currently low in implementation feasibility, 
the measure has considerable value from a conceptual perspective. The 
form of data element ^7 (iii) and its data sources are shown below: 



^ non-competitively employed 
26 closures who have main- 
tained or improved closure 
skills at follow-up (1 year) 
^ ^ non-competitively employed 
26 closures examined at 
follow-up 



X 100 



l5i 
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Data Source 


Data Item Identification 


Follow-up Survey 
Follow-up Survey 


Non-conipetitive work status = same ' 
as S (iii); Closure skills = same 
as-^aia£ul activity .defined in 5 
(iv) 

Select for non-competitively 
^employed 26 closures, using iden- 
tifying data on survey instrument 



Reviewers' concerns with data element 7(iii) focused on the problems 
involved with measurement: 

• The difficulty of defining what constitutes improved closure 
skills make the measurement of this variable problematic. Also, 
it has the traditional prbblems of validity of self-reported data. 

• The nature of the data requires a special follow-up after 
closure. An enormous sample will be needed because of the low 
proportion of non-competitive 26 closures. Who will pay for it? 

• Factor analyze the survey items, and utilize the resultant sub- 
scales as indices. 

• Substitute Walker Associates' Independent Living Scales. 

The main problem with data element 7 (iii) is that, like data element 
5 (iv), it relies on a cumbersome computation based on over 20 questions 
on the follow-up survey. During the data coding process, it became clear 
that many respondents skipped this section of the survey, either because 
it was too long or because they did not understand it. 

Recommendation ' - • - 

Like data element 5 (iv) , we feel that this data element is important 
for evaluation purposes and should be retained in the standards. We 
recommend that the same FAX and LSI data elements used for data 'element 
S (iv) be used here. However, for the purposes o£ this data element the 
PAI -and LSI items will need to be modified into a form suitable for self- 
administration by the clients. The specific itdtns and their forms will be 
determined after completion of the upcoming pretest for the MIS. Until 
that time, data collection for data, element 7 (iii) would be deferred, 
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FINDINGS FROM THE PRETEST DATA 



Table 14 presents the results of the analysis of the pretest data for 
Standard 7. Values for data element 7 (i) ranged. from 24% to 77% with most 
__states falling at the lower end of this range. Further, because these- 
earnings at follow-up figures were not adjusted for inflation, the findings 
may t>e artificially high. In reality, it seems that most clients, for what- 
ever reasons, have not retained :the earnings gain achieved as a result of 
rehabilitation. 

tlie results of data element 7(i) illustrate the concern for reliability 
whichHrrss been repeatedly voiced during the pretest. In all six states, 
approxim:|tely 5% to 10% of the sample had an increase or decrease in earn- 
ings o^^ive dollars .per week'or less, ^Vhile it is possible than, an 
individual could actually incur a three dollars per week cut in salary, 
it is far mor^e likely that the 'Uoss** is a result of rounding error in the 
reporting of earnings. For example, a client may' report his earnings at 
yclosUre .as ''$201.85 per week" (coded as $202), and at follow-up as ''about 
$200/p'6r week" (coded^.as $200). The result of the client's approximating 
his, earnings at* one time and not at another is the appearance of an earn- 
ings loss when in fact there may have been no change, or at best a very 
negligible change. While^ our analysis is by no means conclusive, it does 
suggest that any earnings change o^ less thah> say, five dollars per weelc 
should be considered "no change" for the purposes of this data element. 
The net effect of this change woul^ be to uniformly raise all states' 
scores by a few percentage points\ * . . . • 

The figures for data element ^(ii) range from 1.47 to 7.46, with a 
lower score LndicatlTng better performance.' However, five of the six states 
had ratios of 1.90 or less. The lone outlier (Pennsylvania) illustrates 
the need to report the numeratp.T and denominator along, with fhe ratio. 
This state had a rather high (bi^t\ not the highest) proportion of clients 
relying priijiarily. on public asi^i^t.ance at follow-up (the numerator). How- 
ever, this state had by far the lowest prbportlhsm of clients relying/ ' T\\ 
primaiitly on public assistance at closure (the 3enominator) . If the rep^r.t- 
ing system included only the rsttxo, the performance of this state would 
indeed look quite dismal. By incl^uding the denominator, the finding offers 



Table 14 
Statistics for Standard 7 







State 


Datci Flomont 

Vi' »-* W VI 1 « 4. lit w 1 1 U 




VA 


PA 


MI 


DE 


OR 


I MS 
) BLIND 


71 
« 


# 26 closures with jobs 
at closure who retained/ 
increased earnings ^ 

^ 26 closures with jobs 
at closure examined 
at follow-up^ 


28.6% 


27.6% 


25.3% 


' 24.1% 


50.5% 


76.9% 


7ii 


% 26 closures with public * 
assistance as primary source 
of support at follow-up 

(I year). / 


1.90 


7.46 


1.72 


1.88 


1.68.. 


1.47 




% 26 closures with public ,j 
assistance as primary source 

6'f support at closure ' 






7iii 


^ non-competitively employed 

26 closures who have 
maintained/improved closure^^ 
skills at follow-up (1 year) 


' 79 . 2% 


40.0% 


42.1% 


46.5% 


100;0% 


68.2% 




// non-competitively employed 
26 closures cxamjlned 
at follow-up 


. (n=24) 


(n=2^ 


' <:n=19) 


■ (n=.23) 


(n=3) 


(n=44) 



Date Sources: 1 . Six Month Follow-up Survey 
2 Revised RSA-300 
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a more informative and useful finding of the state's performance on this 
indicator. 

Values for data element 7 (iii), which represents the percentage of 
all non-competitively employed clients who have maintained or improved 

their closuife ski Us at follow-up, range from 40-100%' This percentage 

is computejf^ terms of the number of clients who stated they had improved 
on any 2^pec^x:ef^^lf-care, attitudes, .homeiriaker skills, joli^related skills 
or had "improved so as to release other family members to join the labor 
force." As noted earlier, we recommend changing the methodology used for 
this data element. 
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STANDARD 8: CLIENTS SH?ML BE SATISFIED WITH THE VOCATIONAL REHABILITATION 
' PROGRAiM, AND REHABILITATED CLIENTS ^SHALL APPRAISE VOCATIONAIT 
- REHABILITATION SERVICES AS USEFUL IN ACHIEVING AND MAINTAIN- 
— TNtTTHEIR VOCATIONAL OBJECTIVES. 



Data Elements : (i) Percent closed clients satisfied with overall 
, VR experience 

(ii) Percent closed clients satisfied, with specifi,c 
aspects of VR ' , 

(iii) Percent 26 closures judging services, received 
to have been useful in obtaining their job/ 
homemake'r" situation 

(iv) Percent 26 closures judging services received 
to be 'useful in current performance on the job/ 
homemalcer situation 



As an indicator of consumer appraisal of services, the standard on 
client satisfaction with /ocatipnal rehabilitation^ services has con- 
siderabl-e merit-. Since client satisfaction polls usually offer high 
degrees of support for the program, this standard is 'viewed as having 
distinct political value in lobbying for expanded financial support at 
both the state and federal level- There are problems, however; with 
using such undifferentiated measures for indicators of program effective- 
ness. If all, or nearly all, clients express satisfaction even though 
the program does not successfully rehabilitate them, this measure may 
tell us something * about 1^R, but it won't tell us about the program's 
utility. 

Complementing the political utility of a satisfaction measure is 

.the inclusion of a client utility assessment in the standard.' The 

intent of this -clause is to ensure that successfully closed clients 

If 

zssQSS the utility of VR services positively in terms of actually having 
contributed to their getting a job and functioning in it. As a sub- 
stantive qualification of the satisfaction standard, utility 'assessment 
offers a valuable entree for probing areas needing, program improvement 
and for ensuring consumer involvement in improving the responsiveness 
ov VR services to client needs. * . ^ 




, " The cQtnments and suggestions of reviewers were^ generally appli- 
cable tQ all four data ejleiiients' g,s well as the -Standaijd. They will 
be discussed following the descriptions of eac|i ^ata element. 



DATA ELEMENT 8 (i):' PERCeNT CLOSEP CLIENTS SATISFIED WITH OVERALL VR ' 
■ / / ..EXPERIENCE ■ , •• ■ 

•As one of the clata elements of the original n.ine^5tandards, 
retailing overall satisfaction .as ^ measure of program performance has 
s^ver^l advantages: (1) the procedure is in place; (2) developmental 
costs' have already be^rt absorbed; (3). it constitutes a composite measure 
of client satisfaction which, responds to legislative and consumer advo- 
cacy concerns; and (4) it is viewed as politically valuable information r 
by the states; and (5) the data show some discrimination among closure 
statuses . ,^ 

The disadvantages of such a measure' include several aspects of the 
methods in which satisfaction information is currently gathered. The 
expenditure in time and money to launch follow-up surveys of clients by 
any technique is only questionably cost-effective, and' can be expected ^ 
to have response bias.. Unsucces^sful clients, who are more likely to be 
issati^^^ed with the program, are also far less likely to be located 
ix months to a year after closure and less likely, even when contacted, 
to respond to questioning. As a consequence, an already highly ^ub'jective 
measure 'suffers additional biasingf. ^ It would be possible to mitigate 
some, of these concerns by administering satisfaction questionnaires at 
closure. 

A further disadvantage is the ambiguous nature of the composite 
satisfaction questi(!^n: one client's ''overall experience" may really be 
an appraisal of his/her feelings About the counselor; another client's 
outcome may positively or negatively color the "overall experience." 
Proposed data element (ii) a'ttempts to mitigate this problem and provide 
for areas of inquiry to isolate the correlates of satisfaction. 

An extension of the problem of ambiguity is that of non-comparability 
of data across states due to differences in wording, * sampling,' timing, 
and the individual contacting the client. Ml of these variances create 
substantial' and unidentified biases which could be mitigated by a . , 
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consistent questionnaire and methodology provided by RSA for state use. 

The form of this data element and the sources of the data are as 
follows: 



# closed clients satisfied with 
overall VR ^experjience , ^ 

# closed clients surveyed 



Data Source 



Closure or Follow-up 
Survey 



Closure or Follow-up 



Survey 



'Data Item Identification 



Closed clients = 08, 26, 28 and 
30 



All closed clients surveyed 



DATA ELEMENT 8 (ii) : PERCENT CLOSED CLIENTS SATISFIED WITH SPECIFIC 

ASPECTS OF VR 

Introduced in the original nine standards, this data element attempts 
to mitigate several of the failings of the composite measure, in data 
element (i). In particular, the aspects isolated for inquiry include 
questions about the counselors' protection of the clients' interests, 
the kind and timeliness of service delivery, the results of restoration 
efforts, the job placement process, and the ultimate vocational attain- 
ment. Ideally, this discrete approach to the VR experience would res\ilt 
in mixed responses from clients which in tiirn could lead to more extensive 
probing ^in a face-to-face or telephojne interview, the opportunity for 
follow-up from a written questionnai*re, or, at,^1^he least"; an opportunity 
for the client to air any dissatisfaction in an open-ended question. 
Consistent negative assessment in any one, of these' ^xea.s would be highly 
useful in guiding state evaluations arid providing sut^st'antive input to 
programmatic improvements. The ^rm ^nd data sou^ipes for. this element 
are shown below. ' - - • ; ^' ' t 
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# closed clients satisfied 
with specific aspects of VR 

# closed clients surveyed * 



X 100 



— ^ 

Data Source 


Data Item Identification 


Closure or Follow- 
up Survey 


Closed clients = same as 8(i); 
Specific aspects = client's pre- 
sent situation, information on 
disability and opportunities, 
counselor promptness, physical 
restoration ^ervices, job 
training services, placement 
services 


Closure or 
Follow-up Survey 


All closed clients 



DATA ELEMENT 8 (iii) : PERCENT 26 CLOSURES JUDGING SERVICES RECEIVED 
' . TO HAVE BEEN USEFUL IN OBTAINING THIS JOB?" 

HOMEMAKER SITUAT ION - ' 

7' 

Consumers are not always qualified to judge the techrtical qualities 

of services received, the appropriateness of the training to the outcome, 

or whether the services should have netted a better job. The .client 

can, however, make a fairly objective assessment of whether the services 

he/she received were instrumental in securing the outcome situation. The 

concern captured in this line o£ inquiry differs substantially from 

quesMoning whether a client was satisfied with the services; it looks 
> 

at causality from, the client's perspective, independent of an outside 
opinion as to the appropriateness of the service plan to the attainment 
of the closure situation. 

Utility assessments necessitate surveying the clients, in this 
case the successful clients only. Although initially conceived as a 
follow-up survey, the infortetion desired for this data element may 
actually improve if asked at closure. With two months in the employment 
situation, clients could be expected to .assess accurately the relationship 
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between their VR services arid the situation attained. Again, uniform 
data collection methodology is of prime importance in ensulring the 
validity and comparability of this information. The form of data element 
8 (iii) and the data sources are shown below. 



# 26 closures judging services 
received to have been useful in 
obtaining their job/homemaker 
situation 

# 26 closures surveyed , - 



X 100 



Data Source 


Data Item Information 


Closure or 
Follow-up Survey 


Select for 26 


Closure Survey Fact 
Sheet, or Follow-up 
Survey Fact Sheet 


Select for 26, using identifying data 
, on survey instrument j 



DATA ELEMENT ^8 (iv) : PERCENT 26 CLOSURES JUDGING SERVICES RECEIVED 
' / TO BE USEFUL IN CURRENT PERFORMANCE ON THE 

JOB/HOMEMAKER SITUATION 



Equally as important as VR services' contribution to the attainment 
of the client's closure situation is their usefulness in functioning 
in that position. While not unequivocably objective, the client's 
assessment of whether he/she uses the skills and/or knowledge gained 
from VR services is the closest approximation of the case. The form 
and data sources for this element are as fol-lows: 
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# 26 closures judging services 
received to be ^us^ful in current 
performance on the j ob/homemaking 
situation 



f 26 closures- surveyed 



-X 100 



Data Source 


Data Item Information 


Closure or ?ollow-up 
Survey 


- Select for 2d ' 


Closure Survey Fact 
Sheet or Follow-up 
Survey Fact Sheet 

• 


Select for 26, using identifying 
data on survey instrument 



REVIEWERS^ COMMENTS AND SUGGESTIONS 

The following issues were raised regarding Standard 8 and its four 
data elements. % 

• this is an" important thing for VR to do. It can help us out 
in times of budget battles. 

• Clients respond in generalities about counselor, not in 
specifics.' this may lead to misinterpretation. 

• All of the data elements of Standard 8 raised questio.ns about 
the validity of self-reported data and the high rate of 
non-response . 

• Client satisfaction questions are in need of refinement. Some 
agencies requested this standard not b6 emphasized because of 
the reliability problem. 

• Data source should be the closure survey rather than the follow- 
up survey. 

• The value of information for program planning and management 
purposes is greatly exceeded by the cost of obtaining the 
information. 
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• Drop all data elemertts in Standard 8, and transfer the Standard 
to Procedural Standards. The new Standard should require each 
VR agency to develop' a federally approved mechanism for solioiting 
input on client satisfaction. This consumer, input should Jfe 
evaluated and, where appropriate, used to effect program changes. 

The claim that clients respond in generalities about the counselor 
appears to b^borne out by the facts. Analysis of the pretest data show a 
high correlation (between 88% and 90% vagreement) among dichotomized 
responses to the three counselor satisfaction questions. This finding 
suggests that these are but one question in the clients' minds and implies 
that there -should be oiUy one question on the survey. 

The validity and reliability of the data were mentioned as possible 
problems with the^se data elements. Standard 8 is inherently subjective and, 
like most subjective assessments, will not have the highest reliability and 
validity. However, as one agency pointed out during a site visit, the real 
issue is not subjectivity, but whether the questions are understood by the 
respandents. This problem is more properly, addressed by revision ofi the ^ 
' instrument. At this "point, we should note that in the current political 
atmosphere, inclusion of Standard 8 is particularly vital, and as a 
practical matter, we may have to settle for less than ideal reliability 
and Validity. 

A question was also raised about the response rate of clients. In 
fact, the response rate 'to the closure survey was fairly good (about 60%), 
and fhe questions pertaining to Standard 8 had rather low levels of missing 
data (6% to 13% missing) . The follow-up survey had a lower response rate 
(about 50%) and a few percent more^missing data. It seems clear that we 
should rely on the closure survey for this standard. It already has an 
adequate response rate, and this can be improved by revision of the instru- 
ment. 

It n?ay be true, as suggested above, that the cost of obtaiitlng 
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information on satisfaction and usefulness exceeds the information's value 
for management. However, we should keep in mind that this standard serves 
primarily a political function; it's management uses are secondaiy. From 
this viewpoint the costs are probably not .excessive. 

Finally, it would be inappropriate to transfer this standard to the 
.procedural standards. Client satisfaction and the usefulness of services ■ 
pertains to an agency's performance and, if at al'l possible, should >De . 
reported on an ongoing basis: A procedural standa'rd in this area woula be 
appropriate only if it were impractical to monitor satisfaction/usexulness 
continuously* For all their faults, the data elements of Stand^d 8 provide 
this capability. 

4 

SUMMARY 

th^re were two major problems raised in the proceeding discussion. 

Reliability/validity of the data . This R^X)blem is especially acute 
for several items in data element (ii), The^ three counselor satisfaction 
items appear to be measuring the same thing, since clients are unable or 
unwilling to draw distinctions among them. These three items should" be~^ — 
combined into a single measurement , /and the three corresponding survey 
-questions should also be combined^ In addition, the "satisfied with 
present situation** item has raxher poor face validity, VR ds only one 
of many factors contributj^ to the client's present situation. Those 
other factors are hot something over which VR has control or for which 
VR should be held accountable. This item should be deleted from data 
element (ii)-. 

Non-response rates, iVhil^J^e response rates for both surveys were 
adequate, the closure survey had both a higher response ^rate and a lower 
missing data rate than Hhe follow-up survey. The closure survey is 
clearly the preferred instrument for measuring tjie elements of Standard 8, 

ANALYSIS OF' PRETEST DATA FOR STANDARD 8^ • 4t 

Table 15 presents the results of the analysis of the pretest c^ata 

'■I • . 
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' ' Table 15 
Statistics fox- Standard 8 
(All from Closure Survey) 



Data f.lcmeni 


Hquation 


State 


VA 


PA 


MI 


DE 


OR ■ 


MS 
BLIND 


8i 


closed cases satisfied with 

overall VR experience 
# closed clients surveyed 


85.5% 


83.9% 


73% 


85.2% 


• 

76.7% 


100% 


8ii 


closed clients satisfied with 
specific aspects of VR 
^ closed clients surveyed 


-- 


— 


— 


-- 


-- 


-.- 


^^^'^^^1 ient * s present situation 


74.6% 


67.2% 


53.9% 


63.4% 


65.0% 


84.6% 


Information on disability 
and opportunities 


86.2% 


84.1% 


. 75.0% . 


80% 


70.4% 


96.8% 


•Counselor's jvillingness 
to listen 


89.9% 


90.4% 


82.0% 


83.1% 


82.3% 


100% 


Counselor's promptness • 


91.4% 


86.7% 


82.5% 


80.5% 3 


83.6% 


98.6% 


Physical Restoration Services 


95-. 2% ■ 


100% 


89.7% 


86.8% 


88.1% 


98.1% 


Job Training Services. 


91.1% 


■ 100% 


6^8% 


8i.6% 


77.7% 


. 85.7% • 


Placement Services 


84.2% 


'73.2% 


74.3% 


85.2% 


76". 1% 


88 . 2% 


8iii 


# 26 closures judging services 
received to have been useful 
i/i obtaining their job/home.- 
maker situation 


86.8% 


, 80.5% 


.80.0% 


92 .^% 


'80.1% 


100% 


Siv 


# 26 closures judgifig services 
received to be useful in cur- 
rent perfoimance on the job/ 
homemaker situation 


83.9% 


83.2% 


76.9% 


90% 


82.5% 


100% 
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for Standard 8. Data for this part of the analysis were obtained from the 
closure survey. 

Values for data element 8(i) range from 73% to 100%. The data element 
represents the percentage of closed clients suryeygj who were ^^t±s£i&d with 
their overall VR experience. 

Data element 8(ii) examines the satisfaction of closed clients with ' 
specific aspects of VR services. Satisfaction with the client's present 
situation varied across states from 54% to 85%. The three measures relating 
to the counselor display an interesting pattern. Ifliile there is considerable 
variation across states for each measure, there is very little variation 
across measures within a state.' In only one state does the variation (i.e., 
percentage difference) among measures exceed 7.5%. And, as was mentioned 
earlier, there was close to 90% agreement among the measures, aggregated 
across states. Taken together, these results suggest that the three 
counselor satisfaction measures are tapping a single underlying factor. 

, The three measurements of client satisfaction with VR services show 
considerably more Variation than the counsjslor measures, both across states 
(e.g. a 40o spread for job*training services) and across measures ' (ranging 
'from 5% to 27% difference depending on the state). Ifliile these measures are 
correlated, they do not display the unidimensional pattern of the counselor 
satisfaction measures. 

* Data element 8(iii) represents the percentage of 26 closures judging 
the VR services they received to have been useful in obtaining a job or 
homemaker situation. Values range from a low of 80% to a high of 100%. . 
Data element (iv) is the percentage of clients rating VR services as useful 
in performance in the clients ^"situation. This measure has a similar range 
of values ,(f^oi^ "77% to 100%). \^hile ^these two data elements have a wide 
variation across states, th&rd ds very little difference between measures 
within a state. This ijo doubt reflects their conceptual similarity. 
Furthermore, when data ^from several states are combined, client ratings on 
the two measures are ib agreement 93% of the time. It appears that these 
two data elements, lik^' the counselor satisfaction items are measuring 
essentially the same l^ing. This close correspondence argues for combining 
the two into a single data element. 
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Recommen dations 

• . There was general agreement among the state agencies that Standard 

• 8 reflects an important and legitimate ^concern for VR, We recom- 
mend that this Standard be retained in its present form, 

• Data element (i) : Client satisfaction with the overall 1 VR ^ 

* experience is a good composite measure. It should be^ re- 
tained. 

• Data element (ii) covers three areas: 

- Client satisfaction with his/her present situation involves 
many factors which may have little or nothing to do with VR. 
Because of its low conceptual validity, we believe this item 
should be deleted from data element (ii) . , 

- The results of the data analysis suggest strongly that the 
three counselor satisfaction items are in fact measuring the 
same thing. IVe recommend that these three items (and the 
corresponding questions on the survey) be combined into a 
single measure. 

- The three items measuring satisfaction with services are the 
most useful for internal management of an agency. They are 
distinct in concept, an^ the data suggest they are distinct 

in clients' minds. They should be retained as three separate , ' 
items under data element (ii) . 

• Data elements (iii) and (iv) were so highly correlated that we can 
confidently conclude, that they are measuring a 'single factor. They 
(and their corresponding questions on the survey) should be combined 
into a single data element. 
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V. ■ DATA COLLECTION F ORMS F OR THE PERFORMANCE STANDARDS . ^ 

.. . 1^ 

r - . . . • . 

OVERVIEW ' . • , • 

' y • , 

Four new forms were required to supplement data already collected 
on a routine basis, in order to meet the. data xequirements"~"oJrthit~eight 
proposed Performance Standards. These forms were: \ 

• • R- 300 -Supplement designed to cplle.ct additional client 

information during eligibility determination and' nVRP .develop- 
ment . 

• R>500' Supplement. C5D designed to collect additional client 
information at^l?P&sure. 

• Closure Surveys designed to be distributed at clostire as a 
mailbacic questionnaire, to collect client satisfaction and 
service utility assessments, aijd to collect information on , ' 
"functional and economic-- gains . . ' . 

^ # Follow-up Surveys designed as a one-year post-closure mailout 

questionnaire for purposes of dSirumenting" retention of benefits 

(economic and functional), client satisfaction, and service 
Utility assessments. 

Training ^ ^ ' . ' / ' 

\ group training session, held in Houstoj^, Texas on December 11 ' 
and 12, 1979, was. the formal training provided on the performaAce 
standards* pretest activities. A comprehensive training manual was 
distribUte|d tJ-each of. the NIEU ^r^pr^entatiw in attendance and'reviewed 
sect ion' b\; seciion at that tim^ For each pretest activity, tjie manual . 
provided the followif^g information: ^ / 

• Background an:4 rationale , 

• /Sampling ' ^ * ^ 
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• Scheduling 

• Data collection methodology 
Data processing a^jd reporting 

• Data collection instrument 



Following the trainirig sessian, on-going telephone liaison was pro- 
vided to the MEU's for clarification and ^problenT solving throughout the 
pretext implementation. Tiwstaites requested a<^ditional technical assis- 
tance site visits at 'the conclusion of the data collection effort to 
assist in summarizing thecstates' experienc^^ (Virginia) and to provide 
decision rules for data completion. (Delaware). - 

Both sampling and scheduling for each activitjr were, flexible to some 
degree, and compromises were negotiated with the MEU's to accommodate 
operational realities within indii/idual states. Th[e following table 
reflects the sample size and schedule goals presented in Ho.uston for each 
performance standard pretest form (see Table 16). ' * 

* '{ " • 

Evaluation of the Pretest J ' > , ' 

• , V 

Three evaluation' activities of the performance standards' content, 
training and implementation experiences were cond^tfted during the pretest: 

^ ^ v:" 

• Prior to the formal ^training an Houston, ^phe MEU's received 
• copi^of ,BPA»s v^^orking papers and Final Reports, as well, as 
an individual site visit during which the proposed standards 
systen) was presented and questions answere^. At the initial 
mee^^ in Washington, D.C./in October, 1979, "the first of 
several evaluation' activities was initiated through the distri- 
bution of Pretest Assessment Form ^1. The purpose of this 
activity was*to document the hEU's initial reactions to th"^ \ 
Performance^^andards , their data elements and the sources lot 
^eneratlnj^ them prior, to the pretest. ' . * 

• Tlie,jlouston training session was also evaluated by the^artici- *• 
Dating repr^ntatives of the MEU's on Participant' Evaluation 
and Comment Form f"l. * ' 
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• • " Table 16 
Sample Size Goals for Performance 
Standards Pretest Forms 



Pretest Activity 


Sample 
Size 


Implementation 
Schedule 


' Submittal 
Schedule 






1 /I /RO-l /3l780 


4 /30/80 


(Status 06 and 10) 




* 




R-300 Supplement C5D 


500 ■• 


1/1/80-3/31/80 


4/30/80 


(Status 26) 


300 ' 






(Status 08 ftom Q2 § 06) 


10.0. 






(Status 58 and 30) 


100' 






Closure Surveys 


400. 


1/31/80-3/31/80 


5/15/80 


(Status 26 from C5D above) ^ 


3(f0 






(Status 28 and 30 from C§D^ 
above) 


100 






Follow-Up Sulrveys 








(1-year follow-up on 26 's 
closed 1/1/79-3/31/79) 


100 


1/31/80-3/31/80 


5/15/80 . 


a 

(6-month follow-up on 26 's 
closed 1/1/80-3/31/79 from 
C5D above) 


300 


7/31/80-9/30/80 


11/15/80 ' 

4 



^ The original plan-called for distribution of the surveys (mailing, as 
a rule) on the last day of each of three months during the period 
(i.e., Janua'ry, February, and Marcft, or^July, August, and September.) 
Actual mailing dates varied across states. 
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• Following implenientation of the pretest, the.MEU's completed 
Pretest 'Assessment Form #3 on the Performance Standards to 
record obstacles tp data collection, difficulties with the 
instruments and specific items on them, and procedural diffi- 

,culties in implementing the Standards' instruments, xhe de- 
tailed reactions and comm|rnts obtained during these activities 
are presented in our- Preliminary Pretest Report (November 24^- 
, 1980, available from RSA)\ ' • 

Time Requirements 

The following table presents an overview of the states' levels of 
effort required to conduct the performance standards* pretest (see_ Table 1 
The average investment of the Program Evaluation and support staff was 
132 person days, or slightly over one-half persx>n year to train staff, 
execute data collection, analyze these data, and manage the overall 
effort. The R-30D Supplements, Closure Surveys and Follow-up Surveys 

\ would appear to require approximately equal effort to accomplish, although 
the 'following factors would alter this: 

• The counselors* time required to complete R-300 Supplements 
A5B and C5D is not included; 

• The clients* time required to complete the Closure and Follow-up 
Surveys is not included: * 

• Execution and analysis of the six- month Follow-up Surveys are 
not included in these estimates. 

Analysis and Recommendations 

In^the subjections which follow, we discuss the new data collection 
fotms in greater detail^. For each form, we present the findings on the 
, MEUs' experience with the form during t^e pretest, on their reactions to 
the form, on the sample dimensions achieved, and on the time required (by 
, counselors or clients) to complete the form. We end each section with our 
recommendations for the information items which will be required to imple- 
ment the standards. 

♦ 
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Table 17 

Performance Standards Level of Effort 
Person Days 10/79-8/80 . 

(Numbers add down) 





DE 


MI 


MS - 


- OR 


PA 


VA 


AVG 


* 

General Administration 


21.9 


30.3 


20.3 


12.4 


.64.0 


28.8 


29.6 


R-300 Supplements^ 


21.9^ 


19.2 


12.0 


10.2'= 


36.5 


26.6 


21.1 


Training 


5.5 


8.4 


4.4 


■"ih 


12.1 


7.8 


6.6 


Execution 


15.9 


3.4 


0.0 


5.5 


21.6 


13.8 ' 


10.0 


Analysis 


.5 


7.4 


7.6 


3.6 


, 2.8 


5.0 


4.5 


Closure, Surveys 


25.4 


23.3 


17.6 


17.3 


24.5 


5.6 


19,v0 


1 1 cLXI 1 X 1 1^ 


O m \J 




A 'A 


1 ^ 
I . D 


^ 1 n 
1^ • U 


1 A 


A Q 
H ♦ 0 


Execution 


16.9 


14.5 


5.8 


9.3 


,10.'^ 


1.8 


9.8 


Analysis 


.5 


7.5 


7.4 


6.5 


1 .9" 


2.4 


4.4 


Follow-Up Surveys^s 


27.9 


20.3, 


15.7 


11.3 


34.1 


1 

' 7.0 


19.4 


Training 


8.0. 


1.0'^ 


4.4 


1.8 


5.8 


1.9 


"3.8 


Execution 


19.9 


15.5 


4.3 


4.6 


22.0 


3.3 


11.6 


Analysis 


0.0 


3.8 


7.0 


4.9 


6.3 


1.8 


4.0 


TOTAL 


97.1 


93.1 


65.6 


50.9 


159.1 


68.0 


89.0 


Adjusted total for 


157.6 


124.1 


65.6 


50.9 


198^1 


198.0 


132.4 


estimated clerical 
















supj3ort 

















Includes A+B and C+D. Labor estimates for" R-300 Supplements do ncft 
include the field Efforts to collect the information on the sample of 
clients. These are generated from the A+B and C+D forms themselves, 
cm which counselors indicated the time required- to complete the form 
on each client. 
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Table 17 (continued) 



The Program Evaluation Unit in Delaware elected to collect all of t]^ 
R-300 Supplemental information, rather than train fieldi= staff. All 
other MEU's trained counselors in district offices t-Q* 'collect the 
information (see' footnote c-) ' • . Y[ 

The Oregon agency i:evised its computerized case statistical reporting 
system prior to the- pretest/ and incorporated all of^the supplemental 
R--300 items at, that time. Training of field staff was done shortly 
thereafter. Labor estimates could not be made because the activity 
was integrated in a much larger effort. ' 

' r 

These estimates represent primarily the IZ-month follow-up activity,-, 
since the monthly time Report information represented here ccJvers 
only the year from October 1979 through August 1980, and most of ^ 
the six-month. follow-up effort was conducted following that. 

The Monthly, Time Reports were only requested^ for professional^^.st^ff 
committing at least 25% time to Task-IV activities.' Oregon an^' 
Mississippi, however, provided time, sheets on 'th^ir support staff. . 
Based on iatemal time reporting, Delaware' could document clerical 
investment of 1/4 time over the U months, Michigan estimated that* 
an additional 1/3' of the professional time spent. on Task IV was 5pent 
by clerical staff. Pejihsylvania provided very, specific estimates of 
the clerical investment, 'differing from month to' month depending upon 
the nature of the Task IV activities and amounting to 39. person days. 
Virginia estimated support staff involvement >ased on internal time 
reporting at slightly greater than a half-time* commitment. 
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R--300 SUPPLEMENTS 

Supplemental R-300 Form ASB was designed to collect additional client 
information during eligibility determination and IWRP development. Form 
CSD is used at case closure for a similar purpose* Given their similarity 
in pu]fpbse and in the training required, t^e combine the discussions of the 
two forms whenever' appropriate. 

While the; specific information items being proposed as necessary 
additions to the federal RSA-300 form was mildly disputed on substantive 
grounds, the major concern was an operational one. Without pretesting 
the validity, reliability and utility ^pf the proposed addition^ prior to 
requiring massive revisions to state R-300 data systems (forms, methodolo- 
gies, data processing and reporting), the states might well incur major 
costs, delays and st^tff dislocations which could then be deemed unnecessary 
pending the analytic results of the pretest data. This concern was fore- 
stalled by the development of the R-300 Supplemental A§B and C§D forms, 
designed to collect the proposed ^ditional information on a time-limited 
sample of appropriate .cases only. 

Example copies of the fR- 300 supplement forms appear in^Exhibit 1 (A§B) 
and in Exhibit 2 (C§D) . It is important to remember that the supplemental 
forms were developed solely for the purposes of pretesting the data items 
bein^ proposed as ad^tions to the federal RSA-300 form 'and are not to be 
viewed as^ additional instruments for national implementation. The identi- 
jfying information at the top of the forms- was to enable linkage of the 
supplemental information on each client with the states' R-300 data tape. 

The purposes of the individual data items on the supplements were as 
foUovTs : • - \ ' 

■ FORM A 

Item U To* calculate Standards 4(iii) and 6(i). 

Item V For use in comparability adjustments. . 

Item W For use in locating clients during service provision 
and to improve follow-up survey response rates. 

Item'X To calculate pretest level of field effort. 
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SUPPLDJfiNTAL U-3()() rOilM A n 



(To be uscU exclusively for the pretest of the revised VR Standards,) 

I 



Card Nunb^r: 
Agency Code: 

A 

District Code: 
Case Sunb'er: 
Social 'Security Nunber: 
Client Name: 

A rsUPPLSMENT TO R-500 Part 2) 
<T0 BE RECORDED AT COMPLCTION OF REFERRAL PROCESS) 



If closed fron Status '02, complete Form D,' 
Supplecent to R-300 Part 3, If accepted- 
for Extended Evaluation or VR Services, 
complete Fom A, Supplement to R-300 ^ 
Part 2. 



2 / 

/////'/ 



/-/ 



/-/ . / / 



EXHIBIT 1 



1 

2-3 
4-6 
7-16 
27-25 



U. Earnings Year Prior to Referral r $ 
V. Federal Special Program Identification: 



/,/ 



\'one ITF 


'.£T 






PC 




SEC4IS? 


SO 


1 


















000 IJOl 




004 


010 


o:ojo4o 


100 


200 


400 



Sum: / 



W. 











Permanent \ddress of Friend or 
Living in State: 


Relative 






(if availabj^e, enter 1/^ 
if not, enter 0]) 




/ / 




Last \ane First Name 


Initial 






Address: Street and Number 








City State 


Zip Code 







Area Code Tel^jphone Number 
Relationship to Client: 



X. COUNSELOR: Time in Minutes Required 
to Complete Form A: 



2^r30 



31-33 



34 



35-37 



FORM 3 (SUPPLC*X\T er:T;-.Tn\ R-50Q Part? 2 and 3) 
(TO BE RECOROrD AT CCMPLF.TIO.V OF IWRP) 



Conpletc on all clients accepted for 
VR Services 



A. Oa:c f'.vI^P Conpletcd: 

B. Work Status of Orijjmal IWRP Coal: 

C. Oriqmal Cccupationnl Coal (Title): 

COT Code: - 

CCUNSHLCK: Tine in Mi npt cs Required 
to CoTjplete form B: 



/ / / - / / ' / / / 

month day year 

/ / , 



/ ./ / / / 
I'll I 



38-43 
44 

4S-i3 
49-51 



Counselor Signature 



/ 

/ 

/ 



167 's EXHIBIT 1 (cont'd.) ^ 



nrriiiTios'S to ur usm in cr)'iPi.r.Ti>:n suppLCMi:NT\t. R>5on formNa f, a 

f 

Idcntif'/ing Infornation '( ' 

Agency Codo: jf-wo-digit code from RS .^lanual indicating pretest state' agency , as follows: ^^""^^ 

\' ^ 06' De^l aware ' 36 Oregon 

^ 21 Michigan 57 Pennsylvania \ / • - 

73 Mississippi Blind' 46 'Virginia ^ ^ 



District Code: 



Three-digit code using whatever district nunber is currently ass/gned if the 
state agency. Right lu^tify^' • * ^ 



Case NUTber: • * Up to. ten-digit code which should correspond to the client i46ntif ication number 

u^ed in* the state agency for R-500 coding purposes. Right j'ustify. 

Social Security N'o.: Nine-digit Social Security Nunber. For thd purposes of th6 pretest only, tp 
facilitate lipkage with other data sources. . / 

. / ' 

Client Name: ^ ' .For the purposes of the pretest only, to facilitate linl^age with other data 
sour<ves. ^ / 

* FORM ^ rg^^oplenent to R-3Q0 Part 2) ' ; 
U.^ Earnings Year Prior to Referral: ' 

/ / ^ ' Enter the anount (to tf\e nearest dcfllar) the individual earned in the year prior 

' - . -^f date he or she was" referred. If this /client had no earnings, enter "O". 

This iten is an estimate provided by the in^vidual of his or her cash earnings. 
Use the sane definitions. curre:\tly used in/the R-500 Part 2 Item P for weekly 
, . earnings. RigKt Justify. 

y.'. Federal Spocial Program Identification: 

* ^se the s,aine* categories and definitio^ns of them as currently used in R-300 Part 5 
** Iten^D. 

W. ^Permanent Address of friend or Relative Living in'y^tate: 

Provide the name, street^addres^s, o*ty, state, tip code, telephone number, and 
^ ' ^ relationship of persqn to cli^n^ for the purposes- of facilitating closure and 

follow-up contacting, , 

Counselor- Time Required to Complete Form/A 

y The counselor completing the form should estimate his oiTlIer level of effort in 

^ minutes for collecting and filling out the inform^tioji/requested. Right justify. 



FQR:>! 8. fSt:fp I C'nea.t ^ef.>een R-300 PartV2 and' 5) 
A. Date IWRP Coiij)ieted: 
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. - "The date tfv^^client»s original IKRP, after acceptance for VR sei:vices (Status 10), 
was conpl<rted. \ 

B. tv'or^ Status of Original IWRP ,GoalK \/' ^ ^' ' ^ ' 

, " ^'he same one-digit categories and *thpir definitions as currently used in R-300 Part 3 

*' ' . It^'^ J» but describing what the client will be doing if the plan is euccessfully 

» . completed: . , - ^ ' ' 

1 Competitive Labpr Vini'ket 4 Stnte-agency-managed BEP 

' , *2 Sheltcrc<i Workshop 5 Homemaker ^ 

\ . 3 Self-employed (except BEP) 6 Unpaid Family Worker 

- .. ■ 

^ . C.v •Orit:i'nal Occupntiooal Coal (Title): ^ . . ' >^ 

f • • 

J ^ purposes of checking the classification of the uork status in B.B. above > provide 

. the actual job i;oaI listed m the oritjmal IiTRP; » ' 

rCT CoJc^: Record four -dig it code from the irictionar*. of OccututxQrtal Titles t'(jr the vocational 

goal stated in B.C. above. ] ] ' 

'0. £ Counselor: Tine Required to ^Complete Fora B: ■ , ' . • 

r ' * (Same as Tom A.X. above) The counscj'lor^^ronivleting tho .I'oro shmiUl e.-itimate hrs or'hcr 

- level of cffaKt in mimit*>«; fnr coHectinq filling out the infornarion. requested. , 

Rl|;ht justify. » , , . - ' 

. ■.: . ■•, ■ . 

.A 
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SUPPU'MCNTXL ^^500 FORM C D 

(To be used exclusively for the pretest of the revised VR Standards.) 

Card Nunber: ' 111 
« 

Agency Code: • III ' • 

District Code; I I J I 

Case Vunoer: ^ I L I I I I I I /' / 

Social Security Nuniber: III 'f-l / /-/ / 1^1 I 

Client Nanje: » i • . 



FO^M C rsUPPl^MEST 3£r>;BHN >R-500 Pa rts \tx y) 
(TO 3E TAKEN' FROM COlPlETED IWRP) 



.Co-.plete oaly on closed clients 
accented tor VR services. 



A. Date I'.^RP Coroietco- / / ^ / / //__/ 

non^r. dav Vear 

»• ' 

B. Work Status of Ori^:iral I'.vRP.Goal: / I 

C. Orisir.a.l Occupational Goal (Title) 



DOT Code: I I f. I ! 

D. i:Oir.'SELOR: Tine m Minutes Required 

to CoT.Dlcte Form C: ' . II ^ I 



FO^M D r-S>JPPir^'ENT TO .^-300 Part 5 ) 

(TO BE RECCPD5D AT TP:H OF CipSUPsc) 

Cor.olete on closed clients reaching at 
least Status 02, inch/ding 08 » 26, 23, 
and 30. / 
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EXHIBir2 (cont'dQ 



PPFISMTIOSS TO or U5FI) IN COMPtrTIvn $»PPLr^C^:fA!. R«500 FORM C f* D 
Identifying Infor'''nt '.on * 

AgcnVv Code:- • Two-digit code fron RS Minual indicating pretest state agency., as follows: 

06 Delaware * 56 Oregon 

21 Michigan 37 Pennsylvania 

73 Mississippi Blind 46 Virginia 

District Code: ^ Three-digit code using whatever district number is currently assigned in the state 

agency. Right justify. 

Case Nunoer: \ Up to ten-digit code which should correspond to the client identification nuaber used 

in the state agency for R-300 coding purposes. Right justify. 

Social- SecurUy No\ Mne-digit Social Security NHimber. For the purposes of the pretest only, to facilitate 
linkage with other data sources. 

Client Nane: * . For the purposes of the pretest only, to facilitate linkage with other data sources. 

V 

FORM C fSurgleTtent Bef^een R-500 Parts 2 and 5) • . 

A. Date I>.RP Conpleted: ^ * \ * v 

(Sare as Fom 3. A.) The date the client's original Il\R?, after acceptance for VR 
services (Status 10), '«as cor.pletea. 

B. work Status of Cr:zinal IWRP Goal: 

(Sane as rorn B.B.) The sane one-diait categories and their definitions as currently 
used in R-300 Part 3 Ite-n I, but describing what the client was to be doing if the 
plan were successfully completed: 

1- Conoetiiive Labor Varket 4 State-agency-nanaged BE? 

2 Sheltered **.orkshop S Honenaker 

3 Self-erjployed (except BEP) ' ^ Unpaia .«^ami I y Worker 

C. Orig;ral Occupational Goal (Ti^le): * , 

' (Same as Vorn B.C.) For purposes «f checking the classification of the work status 
in B.B. above, provide the actual ]ob goal listed in the original IWRP. 

DOT Code: (Sane as Forn B.C.) Record four-dT^it code from the Pictionarv of Occu^tional 

Titles for the vocational goal stated in C.C. above. ' * * 

0. Counselor: Tine' '"equi red to Complete Form C: i 

(Same as Fom A.X.) The counselor completing the form should estiriate his or her 
level of effort in minutes for collecting and filling out the information requested. 
Right justify. • ^ 

FOPM D ^^u'^nlO'-ent to n-*300 Part 5) 

Entei* the code given for closure status. ' - ^ 

Q. Prirnrv Source of Support: j ^ * 

The same tv.-o-digi^ categories ''and their definitions as currently used in R-300 Part-2^ 
Item S, only capturing the primary source of support 'at closure: 

00 Current "earn;m;s, interest, dividends, rent OS Public institution,' tax supported 

01 Ft^iI)- and friends ^ ^ ' 06^ Woi^men's compensation 

03 Private relief ngency ' ' 07 Social Security Disability Insurance benefits 
- 03 Tuolic assist.incc, at least partly with \ OS All other public so.urces • ^ 

fcdorol tund<; ' 0^ Annuity 6r other non-diiabi 1 ity insurance 

04 FuDlic a^3:star.co, without federal funds , benefits (private) 

• (Ox only) • 10 All other sources of support 

R. Nupocr •>:' I'.O'iT", '..orkm? Per Week j ^ ^ 

^ntcr the n'nber of hour*; the individual is 'working per t^ek. If the client is a 

honeniker or is not worNin^;, enter "u". This ite'n is an estimate provided bv the 

* client. Ric:iit justify. , ^ ^ » 

S. Couns^el'or': Tine i>cquired to Complete Form I): 

(Same ns Tom A-.X.) The coun^c^lor completing the fom should estimate his or her 1 eye 1 
o.f effort m minutes for collecting .md filUrn; out tho inform.ition^xcqiiC5toJ. Right 
justify. ' " 



FORM B and C (contained identical data items) 

Item A Pretest^ validity checkpoint. 

I^m B To calculate Standard 5(iii), 

Item C Pretest, validity checkpoint. 

Item D To calculate pretest level of field effort. 

FORM D 

Item Q " To calculate * Standard 5(ii'). 

•Item R To calculate Standards 4(iv) and 7(ii). * • 

Item S To calculate pretest level of .field effort. 

DATA COLLECTION NffiJHODOLOGY 

The data collection methodology stipulated that state ^staff 
would train counselors in all or a sample/of state districts to collect, 
the supplemental information., For the A§B form, the -data collection plan 
.called for counselors to collect the ASB data on all or a sample of 
clients at acceptance (for VR services or extended evaluation) and ^t 
completion'of the IWRP (for those -clients accepted for VR services). 
Since the proposed sample size was only 100 clients in Statuses 06 and 10 
during the month of January, 1980 and the states ' estimated volumes of 
such clients ranged frpm less than 100 to nearly SpOO during that time 
perio'd, the states employed variolis approadies ta^)ecuring a representa- 
tive sample. Table 18 illustrates the sample siz^d ^chieved in each MEU, 
the proportion of state districts and the numbei" of, field staff involved 
in tne data collection effort. ^ 

-> \-nx the C^,D form, the- '^ta collection methodology stipulated that 

the state staff would train counselors in all or a sample of state 
districts to collect the supplemental data on all or a sample of clients 
closed during the first quarter of 1980. Sinc'T^tHe.^U volumes of such 
clients ranged from well below to well above the sample goals of 300 
26-closure5i, lOjl 28-and 30-closures, and lOO^-closures from Status 02 
and the states einployed various approaches to securing a representa- 
tive sample. Table 19 illusl^rates the sample' size achieved in each MEU, 

> * . • 

the proportion -of stat9f*districts and the number of field staff involved 

' ' ' » ,* t ' * ( 

in' the data collection effort. , 
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Table 18 

R-300 Supplement A+B: Sampling Dimensions 





DE 


MI 


MS 


OR 


PA 


VA 


Final Sample 
(Goal: 100 Status 06' s 
- and Status 10' s) 


111^* 


' 161 


^ 47^. 


100 


IGO 


161 ■ 


Number of Districts in 
State Involved/Total < 




• 4/42 


17/21 


32/32 


,1/15 


20/61 


Number of Field. Staff 
Involved in Data 
Collection • 


37^ 


41 


46 


275^ 


29 


90 



^All clients accepted into Status 06 and 10 during the sample month. 

^Minimal involvement of field staff sCace PE Unit completed forms. 

^All counselors throughout the state .routinely collect this information. 

Sample Representativeness : 



DELAWARE 



MICHIGAN 



>JJSSISSIPPI 



OREGON 



Sample representative regarding ratio of 06 to 10 c^ses 
appears satisfactory in that it was estimated that Status 
10 cases should represent at .least 75% of total. 

Despite the size^f the sample and somewhat arbitrary 
selection 'critel^i we believe that the clients selected 
for participation in the pretest pf A+B are generally 
representative of MBR clients. 

Northern portion contains twice as many* as central and South, 
which are more populous. Sample consisted of all clients 
accepted into" 06 and 10, but it was only half the requested 
sample size. Only a smaller number could be more evertly 
distributed. 

Not applicable. We sampled to get 100 rather than use 1/80 
cases. ^ ■ 



PENNSYLVANIA Yes. ' ' — - ^ 

VIRGINIA .Representative of volume of clients during the same .pe'riod 

of time in 1979 Regions ][ and II, ^ 
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Table 19 

R-300 Supplement €+D; Sampling Dimensions 





DE 


MI 


MS^ 


OR 


PA 


VA 


Final Sample 














(Goal: 300 Status 26' s 


142 


333 


86- 


.300 


300 


301 


100 Status 28 *s 














and 30 's 


57 


189 


26 


100 


101 


118 


100 Status OS's 














from 02 § 06) 


112 


179 


303 


100 


102 


156 


Number of Districts 


3/3 


8/42 


20/21 


■32/32 


3/15 


19/61 


^Involved/Total 












Number of Field 


37b 


83 


' 46 


275^ 


9 


90 


Staff Involved 















All clients closed in Status 26, 28, 30 and 08 from 02 and 06. 
'^iMinzmal involvement of field staff since PE Unit completed forms. 
"All counselors throughout the state routinely collect this information. 



Sample Representativeness : 



DELAWARE 
\ 

'MICfllGAN 



MISSISSIPPI. 
OREGON 

PENNSYLVANIA 
VIRGINIA 



Sample representation for cases closed 28 and 30 seems fair. 
However, about 66% of the total FY r'ehabilitants are closed 
during the second half of the FY. 

The district offices selected for participation in the 
Supplemental C+D pretest comprise a stratified random sample 
of all MBR district offices and the clients about whom 
. information was collected are believed to represent a good 
cross-section of^-^all 1st qtr. MBR clients. 

All districts in state represented. 

Not applicable. Inappropriate for sample size requested. ^ " 
Yes. . , * 

Representative of volume of clients during the same period ■ 
f of time in 1979 Regions I 'and II. \ 
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Sununary of Pretest Experience 

We have detailed the comments and reacti6ns received from MEU and 
oth^r state and RSA reviewers, in our Preliminary Pretest , Report . Those 
comments pertain to the training. and data collection methodologies^ to 
the instrument formats, and to the individual data items. In thi,s section, 
we summarize tho.se commejitS" andi-reactions . 

Despite the variety'of .caseloads and tfie /different methods used to 
derive a sample population, the MEU's genera/ly felt that the clients selec- 
too for participation in the pretest were rOTresentative of theiY agencies' 
c.ascl-oad. The only concern expressed was that more clients are closed during 
the ?^econa%haif of the fiscal year, a fact not reflected in the C^D popula- 
tion given the sampling time frame. • . . • • 

/ Each state district participating, in the pretejSt sent counselors and 
administrators to a training session to learn how to use the R-300^- supple- 
mentary materials. Only, one state had to augment the existing instructions^ 
for extra clarity. This state also recommended a follow-up session for 
feedback. * 

Data collection was carried out primarily ]>y counselors reviewing their 
case files and contacting clients as necessary to supply missing information. 
In some cases the Planning and Evaluation Division stepped \r\ to relieve the " 
burden on counselors. One state did not have to supplement current data cpT- 
lection techniques due to a recent revision in its computerized reporting 
system, which incorporated all supplemental information into the regular 
reporting system. 

None of the states mentioned any major problems with the* format of 
the data , col lection instrument. The typical response wafe ''''adequate . Some 

r 

concern was expressed regarding the additional paperworj^ involved. 

-Opinions about the procedures to train counselors and administrators 
in usint; theJ^-300 supplements were mostly positive. One -^state -sug'gested 
that the procedures were not' clear and needpd simplification. Another rdised 
the issue of confidentiality of information, as it relates to client privacy. ' 

To improve data collectioij methodology^ one state recommended ;simpii fy*- 
ing procedures and taking the responsibility Q.f data collection otit^Of the 
hands of the counselors. Another state recoingiended^ chanj^ing the format pf : 
the R-5'riO supplement to facilitate clcYieal processings f » ^ 
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General Data Findings Froip^the Pretest 

In addition to time investments required by ^states to ^Implement the 
pretest, other concerns include the amount of missing data found on the 
forms, and the time required by counselors to fi,ll out the forms. The 
reasons for these concerns should be obvious: as regards missiitg data, 
it is little use »to require additional, da^a collection if, despite 
counselor's efforts-, the yiformation is so. difficult, to collect that 
r^"rg^ amounts of missing data rjesult, *As r^ards counselor time require- 
ments, it again is of little use' to require ^additional data collection if 
tha^ level, of. effort is so great as to overshadow the value of the ^addi- 
tional information. Below we present out* findings with regard to the 
levels of^i^ssing data on the A§D and C§D forms, and' on' the time required 
.of counselors to complete the forms. 

,\ Mi^ssing Data Levels ^ . , * ^ 

< V * : — f » 

./ Fqym A The items on Form A had relatively few missing data prob- 
lems. . One|state nas missinfg a third of its data on earnings in the year 
prior to referral • Aside from this, however, there was.lj.ttie or no data 
missing on 'any other item on the form. ^ ' ' . • , • 

. Form^ B The comments of the MEU's suggested that there were 
pr6t)lems in determining the date of IWRP completion. The wide range ^ in , 
missing data rates, (from 2% to 62%) iildicates the large variation among 
the states in their* Ability to collect this datum. ,The variatiori.was ' 
similar for the other items on this* form. The work status of^'the original 
IWi^P goaj, ?he original occupational goal, and the time required to com- 
plete the £om were all missing between 2^a'nd 64% of the time. 

» As noted, onfi^^a'jor problem (both for those collecting the data and 
for those attemptitig to analyze it) concerns the time-frame foj^ata 
collection vis-a-vis the speed with which clients .proceed through ,the 
various statuses. For example; given ^ the short time-frame^, it was quite 
likely that a certain proportion of clients would iiot, h^rve reached status 
J2 by, the end p£ the data- <5bl lection period. Such ciiertts. o'jDVFous^ly (jould 
not^have th^J. part B information recorded at the , point of the state '^s 
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submittal of its 'da?ta. Alternatively, some clients may be closed 08 from 
06, or may be closed 50 from ^$tatus 10, again precluding completion of 
part Unfortunately, thef^orm failed to provide for such situations. 
As a, result, when looking at the amount of missing data on form B, our 
•.interpretation of the findings could be biased toward a judgment of 
"failure to , locate" th^ data. In fact, the true situation (for many 
clients) may be one of "inappropriateness"; that is, the data is jlisti- . 
fiably missing. This problem could have feeen avoided by including a 
space on* the form (to be keypunched) indicating if a client had or had 
not reached status 12. A 

Form C The information on Form C is identical to that of Form B. 
Tlie only difference between them is that the forms are used at different 
points in the case process. This apparently had little impact on the 

encies* ability to collect, the data, IVhile the variation among states - 
was smaller 'for >'orm C (e.g.; 0.2% to 34% for IWRP date), the overall 
levels of missing, data *Kere about the same. 

Despite thes^j findings, it is* possible that missing data levels 
could go down, if part B (C) is integrated into the R-300, In the pre-: 
test, mpst of the clients on whom a C5D form was filled out probably 
reached status 12 long before the beginning of the protest. Thus, part C 
required tlfe counselor to "think back" to when the yC|TF"was completed, 
and what the p-fan said/ (We'll ignpre the fact that the IWRP is supposed 
to, be in th^ case record, dated, and complete^) The same problem exists 
for form B, arid is 'compounded by the "inappropriateness" issue. However, 
if the information was Included as a'separate .section of the R-300, to be 
filled out at completion of the IWRP (as was the. original BPA design) ^ , 
the problem of missing data might'Be greatly reduced. Presuming, of course 
that the R-300 is updated upon reaching status^ 12, 

Form D Tliere are four' items on Form D. The client's closure 

-. - . « 

; ' , ' * 

status was almost always filled in. Similarl>;, the time requirement data 

were mi,ssing on^v- occiasionall^y (betw'den 0% and 6% .of the time). However^ ^ 

problems did arise with the\ other two items. Between \% and- 21% of the 

data on clients' primary source of support was-missing. Finally, the ; 

number of hours-forked per wee*k was missing between 0% and, 43% of the time. 
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Counselor Time Requirements 

Here "ue describe the amount of time required for counselors to complete 
R-500 supplement \^ and C+D. While the figures cited below are informative 
in themselves, it $hould be noted that completion of the forms may take 
less time once they are incorporated into the R-300 form. Table 20 presents 
the average time and median time to complete the various forms'^ and shows 
the number of cases on which data were available, Oregon has already incor- 
porated the supplemental information into its data collection procedures; 
as such, it was inappropriate (and essentially impossible) for the state to 
determine the time required to collect the extra data. 

^ Form A . As shown m Table 20, the average time required to collect 
Form A supplemental data ranged 'from 3.7 to 6.5 minutes per client. The 
maximum amount of time required for any case was 20 minutes; the minimum, 
one minute." As indicated by the median values, for each state it was 
possible to collect the data for at least half of the state's clients with- 
in about five minutes. 

Form B . In the aggregate^ Form B took considerably less time to com- 
plete than did Form A, with average time^ ranging from 1.9 to /3. 9. minutes . 

Form C > The data in Table 20 show that, on average, less than four 
minutes was needed to complete Form C. In data Jike this, the arithmetic 
means tend to be inflated by the few. cases which require a large amount 
of time (cis much as 18 minutes on Form C) . In fact, the majority of forms 
were completed in less than three minutes, and 88% required five minutes 
or less. Note that, often. Form (? required more time on average than did 
the identical Form B. This may result from counselors having to *'lpok 
backj' in time for Form C clients, (who were closed from VR during our sam- 
ple period), whereas the Form B-data, may have more often been completed 
at the- same ti^ije as completion of the IIVRP, thus requiring no "looking baCk." 

Form P . The data for Form-D are very similar to that of Form C/' A 
few cases took a lot of time (up to 20 minutes), while most required less 
than three, minutes . Minety percent of the forms were completed in five # 
minutes or less. 



Table 20 

Counselor Time Required to Complete R-300 Supplement 

(in minutes) 
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4.2 
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2.8 ' 
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' 2,3 


1,3 
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3,8 


3', 9 
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3.4 


2.7 ' \ 
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4 



Oregon had'^incorporated the supplemental information into its R-300 equivalent and"thus could 
not provide data on time requirements. Jt , ' , 
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Suggested Revisions t.o the' R-500 Supplements 



f Below we discuss the data items wjtiich will need to be added in revising 
the R-300. The reader should fecal.1 that these itenfe will become a part of 
the .R-300, to be collected at the appropriate milestone (i.e., acceptance, 
completion of plan, and closure). They will not take ^e form of "supple-* 
menfs*' to the R-300, which will require recontactin^g clients. The recommend 
ations are as follow^ 

» 

Form A (mformation collected at acceptance) 

• Item U: Earnings Year Prior to {leferral . This . item was needed 
to compute 4ata elements 4(i^) and 6(i). We have recommended . 
that both of these data elemehts be removed from the standards. 
Further', for the new data element 6(1) average change in ^ - * 
earnings — * which replaces data element 4(iii), we have recom- 
mended using R-SOO data until implementation of an* annual SSA • 

data link. Thus ,f Item U will not be, -needed 'for the standards^ 
.- ^Recommendation: Delete. Item' U. , * 

• Kem'V: Federal Special Program Identification . This item Would 
be used to compute- -a; "percent severe" statistic. IVhile not 

used specifically ^or a standards data element , _.this piece of 
information cpuld be useful to program njanagers for monitoring ; , 
their caseloads. (The utility of this item will.be further en-, 
hanced if the MI5 spjit R-300 recommenda'ifeion is implemented. 
. Under' this recommendation, R-300 data.ywould be ilnput into ^the 
MJS at the point ttf plan completion. This would include the 
' ■s.ectTdfis qn referral information (sections 1 and 2 of the R-300> 
and on the plan (the new sectidh 3 of the R-300).) Recommendation: 
Retnin this it<|T^^ 

• Item V/: ■ Address of Friend or Rotative . Tjiis item also ts not 
used specifically to coijipute any data elements. However, it ^ 
poteatia41y could be very helpful in boosting response rates to 



— - ; . ■ , 

The new data element l(ii) s.everely disabled as a percent of 
all clients served \^ill be computed using data^from the new RSA-^llS -^ 
form. I . * ' ^ * ^ , 
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the Follow-Up Survey, as well as in keeping contact with clients 
in service statuses ♦ However, it is not necessary or desirable 
for this information to be included on the R-300 input data sent 
to RSA. Instead, we strongly recoirimend that individual state 
agencies include this information on their R-300 equivalent only. 
Recommendation; Delete Item W from the R-300, and encourage 
states to obtain the information on their R-300 equivalent form. 

Item X: Counselor Time . This was needed only for the pretest. 
Recommendation; Delete this item. 

(New item, not pretested) Functional Assessment/Life Status 
Indicators (assessed at acceptance ) . The FAI/LSI items will 
soon undergo a preftest prior to their inclusion in the MIS. 
The' items chosen for inclusion should be added to the R-300, 

r use in addressing data element 5(ii) and the new recommended 
data element 6(ii). 



Form B, Farm C (information collected at completion of plan) ^ 

Each of the items (except D, ^^Coun^selor Time") pertain to the original 
IIVRP. Item B, 'MVork Status;" was needeS^o calculate data element 5(iii). 
Although we recommend moving that data element to the Procedural Standards, 
the, form of the data element remains the same. Thus the information is still 
needed. The other two items Item A, "Date IWRP Completed," and Item C, 
"Original Occupational GoalV are not used to calculate data elements. 
However, both of these items are potentially useful to program managers, and 
we feel they should be retained. 

The date of IWRP completion vis useful to have on the R*-300 both to ver- 
ify the Case Record and to calculate the months a client is^ in status 10, 
The occupational goal is useful to have as a cross-check on work statjuS of 
goal and to inform Jnanagers of the range of occupational go^s made by clients 
and counselors. These itebs must be filled out at the point of completing 
the .orit^inal TWRP 'goal. ' Reoommendation: Retain items A, B, and G. Delete 
item D. • . ' 



>^ "• • ' • • 180 •. 



Form D (information collected at cl03urej 

• Item Q: ' Primary Source of Support at Closure , This item was 
nfeeded to calculate .data elements 4(iv) 'and 7(ii), We have 
recommended delating aata element 4(iv), but retaining ^^(ii) , 
Thus this. item continues to be needed, though the methodology 
for determining primary source of support should be changed 
(as discussed under Standard 7). Recommendation: Retain this 
data item and change instructions for computation of primary 
source of ^support. ' ^ " ^* 

• Item R: Nxmheiy of Hour.s Working Per Week . This item Was heeded 
to calculate data element 5(ii), which has been retained in a 
modified form. Recommendation: Retain Item R*. * 

• Item S: Counselor Time . Needed only for the pretest. Recom- 
mendation : Delete . " 

• (New item,* not. pretested) Functional Assessment/Life Status 

Indicators (assessed at closure). Again, for addressing d'ata ele- 
' ^ 

ments 5(iv), 6(ii), and, 7(iii), client scores at closure on the 
FAI/LSI items must b'e included on the R-30p. . :[ X 

Summary . • • \ *^ 

To recap, the data items which we recommend retaining in (actually, 
adding to) the R-306 are as follows: ' * 

4 At acceptance: ^ ^ - » ' . 

Federal Special Program ID; • ^ 

Address of friend or relative (in state R-300 only no*- 
^ to h6 computerized) ; . 

FAI/LSI items selected after pretesting. 

• At pjan completion: » ' - » |' 
, ' ' Date original IWRP completed.; - • / 

. Work status' o£ original IWRP goal; " , ' 

^ Original bccupati-onal goal . . ' ^ 

• At closure*: . . • ' 

Primary source of support (determined' via methodology 
described under Standard 7); 

o / . ' ■ - • • 

ERIC = . . ^,92 * ' ' 
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CLOSURE SURVEY 

The Closure Surveys were distributed^ at case closure as a mailback 
questionnaire to be completed by fontier clients. The surv.eys record 
client satisfaction and assess the utility of services received. They 
also document functional gains and the degree to which clients attribute 
these gains to Vk services. In terms of functional uses for the pretest, 
the Closure Survey was desired to serve, several purposes: 

• to collect data from clients to respond to sevei^al Standards' 
data elements; • ^ 

• to validate closure information recorded on the R-300, the 
Supplemental C5D Form, and within the Closure Survey itself; 

• to compare' response rates and answers at closure with those 
at follow-up. 

The Closure Survey appears' in the following pages as Exhibit 3. The 
reader can use the Exhibit to understand the purposes pf the specific data 
items, as listed in Table 21. 

Data Collection Methodology 

Tlie data collection methodology stipulated end-of-the-month initial 
mailouts to the sample of closed clients on which R-300 Supplement C5D had 
been -collected (excluding the Status 08 closures). Table 22 iUustrates 
the' sample size, proportion of districts in each state involved in the 
effort, and the response rate achieved by each MEU. The response rates 
cover quite a range, from 46% to 80% return. Not surprisingly, those 
states experiencing higher rates are the ones who followed the proposed 
response rate assurance procedure in greater detail. The procedure. ^ 
recommended included the following steps: v, ' 

• Sending reminder' postcards to non-respondents two weeks^ after 
the initial distribution; 

• MailH^ng a second questionnaire to the remaining non- respondents 
a week. later; 



2*5 Questionnafre No, 

VOCATIONAL REHABILITATION. SURVEY 

CLOSURE QUESTIONNAIRE 

6-7 (1) Which of the following statements best describes your present 
situation? ^Please chock only one box) 

□ 1 I earn a wage or salary at a regular |0b. 

□ 2, I earn a wage or salary in a sheltered workshop (for 

example^ a place like a workshop for the blind or 
Goodwill Industries) 

□ 3 I am self-employed 

□ 4 I earn a wage or salary m a state-managed Business 

Enterprise Program (that is, a state-run program 
mtenSed specifically for disabled people), 

□ 5 I am a homemaker (that is, a person whose primary 
< work IS taking care of the home) 

□ 6 I workon afamilyfarmorin afamilybusiness without pay 

□ 7 I arn not working at present I am: A student 

□ ' 8 I am not working at present I am, Retired 

□ 9 I am not working at present I am, A trainee 

□ 10 Other (Please explain) L 

□ 11 My situation is not listed above (Please explain) 



8 (2) How satisfied are< you with your present situation? (Please 
check one) ' ' 

□ 1. Very satisfied □ ,4 Very dissatisfied 

□ 2 Somewhat satisfied □ No opinion 

□ 3. Somewhat dissatisfied 

9-10 (3) About how many hours a week do you work? (Please check one 
and fill in^the boxes) 

□ 1 usually work about I I I hours a week. 

□ I jdon't know. 

□ I am not working. ^ 

1-13 (4) What are your present earnings? (Please check one and filj in 
. ^ the boxes) 

J_M;3 • □ I earn $ I I I I per week. 

□ I don't know. » 

□ I am working but 1 don't receive a wage or salary. 

□ I am not working. 
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14-15 (5) What Is your largest source of support? (Please check one) 

□ 1, Current earnings from a job. 

□ 2 * Earnings from interest.-dividends. or rent payments. 

□ 3. Family and friends. 

□ 4, Private relief agency (for example, the^ Salvation Army, 

Goodwill Industries 
P 5 Supplemental Security Income (SSI) lor the Blind ' 
Q ^ Supplemental Security fncobne (SSI) for the Aged 
' O 7> Supplemental Security Incorhe (SSI) for the Disabled 
p Aid to Families with Dependent Children (AFDC) 

□ '""9 General Assistance (GA) 

□ 10 Other public welfare (Please explain) 



□ 11 Public institutions (for example, a tax-supported 

hospital, nursing home, treatment center, etc.) 

□ 12. Workmen's Compensation. 

Cil3. Social Security Disability Insurance Benefits (SSDI) 

□ 14. Veteran's Benefits 

□ 15 Private Insurance 

□ 16 Other (Please explain) — — 



16- 1« ^ (6) How much income, II any, did you (or your dependents) receive 
^ last month from all sources of public welfare? (Please check 
' one and fill m the boxes) ' • »-; 

. □ I received $ I I I — I — ' last month. 

□ None 

, □ 1 don't remember. 

20 (7) Sometimes, after receiving rehabilitation services, and training, ^ 

disabled people are better able to care for themselves. This 
may allov. other family members to work, or to work longer 
hours at a job they already have. Is this so In your case? ^ 
(Please check one) 

□ 1. Yes □2. No 0^9 I don't know. ' 

21 (8) Sometimes, as n result of contact with the rehabilitation 

program, disabled people find their personal attitudes change. 
How do you feel about yourself now, compared to befofe you 
were In the rehabilitation program? (Please check one)- 

□ 1. I feel the same about myself. i 

□ 2 I feel better about myself. , 1 (Lo 

□ 3. I feel worse about myself. ^ ^ dL }l» U 

□ 9. I don't know. 

22 m How do you feel about your future, compared to before you 

were In the rehabilitation program'^ (Ple^e check one) 

□ 1 I feel the same about my future. 

□ 2. I feel better ^out my future. 

□ 3 I feel worse about my future, 

□ 9. I don't know. 



(10) Are there any funcllons of telUcare you have Improved In since 
coming to rehabllllatlon* program? (Please go down the left 
side of the list and look at items you might have improved in. 
For each item in the list, check whether you have had any 
improvement and. if you feel you haVe had some improvement, 
check how much you feel ^our improv&rnent is due to help, 
services or training you received through the rehabilitation 
program) 
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23 a Feeding 

24 b Bathroom functions 

25 c Taking fciedicme 

26 d Grooming (shaving. 
* * hair care, make 

up etc ) 
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Talking* 


in2p3n4n5n'6n 


34 


1 


Answering the 
phone 


iD^2n3q4n5n6n 


35 




Moving on the 
street • 


in'2n3. □4n5n6n 


36 


n 


Using public 
transportation 


in2n3a»4n5n6n 


37 


0 
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(11) If you are primarily a homemaker, are there any skills and 
acllvllles related lo homemaking you have Improved In since 
coming io Ihe rehabllllallon program? (Please go down the left* 
♦ side of the list and look at ihe itefns you might have improved 
in. For each iterh in the list, check whether you have had any 
improvement an'd. it you feel you have had some improvement, 
check how much you feel your improvement is due to help, 
services or training you ^received through the rehabilitation ' 
program ) / ^ 
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Washing dishes 
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Cleaning fj^use 
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Doing laund/y 
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Shopping 1or food 
and groceries 
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□ 


5 
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.50 (12) Have you Improved In any job-relal^d skills and activities (like 
typing, using tools and machinery, assembly work, serving 
people, and so oq) since coming to the rehabilitation program? 
(Please check on0) 

□ 1. No 

□ 2. Yes (If yes, please explain the types of improvements) 



5l' (13) How satisfied are you with the Information provided by your 
counselor about your disability and the opportunities for 
lmp«^ipg your situation? (Please check one) 

□ 1 Very satisfied * ^ 

□ 2. Somewhat satisfied 

□ 3 Somewhat dissatisfied 

□ 4. Very dissatisfied 
D 5 No opinion ' 

52 (14) How satisfied are you with your counselor's willingness to listen 

to your ideas and suggestions when developing your rehabilita- 
tion plan and Job goal? (Please check one) 

□ 1 Very satisfied 

□ 2. Somewhat satisfied 

□ 3 Somewhat dissatisfied 

□ 4 Very dissatisfied 

□ 5 No opinion 

53 (15) How satisfied are you with your counselor's promptness In 

providing services to you? (Please check one) 

□ 1 Very satisfied ' 

□ 2 Somewhat satisfied 

□ 3 Somewhat dissatisffed 

□ 4. Very dissatisfied 

□ 5 No opinion 

54 (16) Did your counsetor arrange for you to have physical restoration I 

services such as medical treatment, physical therapy, artiffical 1 
limbs, eyeglasses, dentures, hearing aids, etc.)? (Please j 
check or^) ^ ' ^1 

□ 2 No , I 

□ 9 I don't remember, ' 

□ 1. Yes 

eg A. If yes, how satisfied are you wi^lh these services'> , 

y ' / (Please 'check, one) ' \ 

□ 1 Very satisfied ^ j 

□ '2 Somewhat satisfied * ^ J 

□ 3 Somewhat dissatisfied ' f "^C 
% □ 4 Very dissatisfied * * 

□ 5. No opinion 

u 

56 (17) Did your counselor arrange for you to have )ob training? 

(Please check one) 

□ 2. No ^ 

□ 9. I don't remember. 

□ 1. Yes 

57 A If yes, how satisfied are you with the kind of 

training you received? (Please check one) 
I □ 1. ' Very satisfied 

' 0 2 Somewhat satisfied 

□ 3 s\)mewhat dissatisfied 

□ 4 yery dissatrsfied , 

□ 5- No opinion 



58 ^ (18) Did your^counselor help you Ippk fgf a Job? (Please check'^^ 

□ 9, I don't remember. * " * . y 

□ 1. Yes, ^ * W* . 

A. If yes, how satisfied are you with this help? (PfeaL'\ l 

^ check one) ' \\ 

□ 1. Very satisfied jj 
n 2. Somewhat satisfied 

□ 3. Somewhat dissatisfied ' " '-^^i' 

□ 4. # Very dissatisfied ^ . • *^ V 

□ 5. No opinion • * 

60 (19) How US£^ were (he services or training you received frdm the ' 
rehabilitation program In helping you to get your present 
situation? (Please che ck one) ^ present ^ 

□ 1. The services pr training were of great use 
U-'^ The services or training were of some use*. 
U 3'. The services or training were of no use at all 
U 4. I have no opinion. 

□ 8 I received no serWoes or training.from the rehabilitatiort 

program. 

^^^^ llV'^u^ 

rehabllita Ion program In helping you to perform In your present 
situation (even if they were of little or jjo use In helping yol get 
it?) (Please^check one) ^ ^ ^ 

□ 1. The services or training were of great use 

U 2. The services^ or training ^were of soine use ^ ^ 

U 3. The services or training were of no use at all ^ 
U 4. I have no opinion. 
^ : □ 8. I received no services or training from the cehabilitation 
program. ' 

Ilhlmf f/'^"* ^^^^ experience with the 

rehabilitation program? (P/ease check one) 
,U 1. Very ^satisfied 
, □ 2. Somewhat satisfied 

□ 3. Somewhat dissatisfied 

B3 (22) Wpuld you recommend your vocational rehabilitation agency to 
a disabled friend? (Please check one) 

□ 1. Yes □ 2. No □ 3. No opinion 

64 (23) Who answered this questionnaire? (Please check one and fill 
' ^tn the blank) 

□ 1. I, the former rehabilitation program client, answered 

this questionnaire by^ myself. 
0 2. I ;(he former rehabilitation program client, answered 
this questionnaire with the help of another person. 
This person is'my: (Please give relationship) J ^ 

n : — ^ — ^ ■■ lZZ^uu 

U 3. Another p.erson answered this questionnaire ^or me 
This other person is my: (Please give relationship) 



m 
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H 
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(24) Do you have any comments you wish to make about your case 
or the program, or Is there anything else you think It would be 



useful for us, to ^now_ 



^5-66 
A 



How long did It take for you to flit out th|s questionnaire?* 

I I I Minutes » . . ' 

(26) Do you wish to be contacted by your vocational rehabilitation 
agency for fy^her services^ If so» please provide:, ^ 



Nam« 




1 






Oty 




Slat« • 




Are* Cod* 




^ T«l«pTion« Numt>«r 

V 





00 
--a 



tn 



03 
H 



V 201- • 
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Table. 21 ' * < ' 

Purposes of Sp'ecifrc Closure -Survey Items 



Item(s) 


Purpose (s) 


l' 

» 


/ Pretest validity checkpoint of R-300 

Part. 3. I, for Standards S(i) and 5(iii) 


3^ 


Pretest validity checkpoint of R-300 
Supplement C+D Item R for Standar<l^ 5(ii) 


4 


Pretest validity checkpoint of "R-300 
Part 3. J. for Standards 4(i) and S(ii) 


5 


• 

To calculate Standa'rd 4(iv) and to validate 
R-300 Supplement C+D Item Q 


6 


Pretest validity checkpoint for Closure 
Survey Itenr 5 ^ ^ 


7 - 12 


To respond to Standard 5(iv) 


2, 

13 - \Z\ 


To respond to Standard 8^ii) and as pretest 

validity crosscheck with Follow-up Survey 

Items 2, 16; 17, 1-8, 19A, 20A, 21A 
* 




T5 respond to Standard 8(iii) and as pretest 
vdfclidity crosscheck, with Follow-up Survey 
Item 22 ' 


20 

\ • 


' To respond to Standard 8(iv) and as pretest 
validity crosscheck with Follow-up Survey 
Item 23 ' • . 


21 " ' _ * 


To respond to* Standard 8(i) and as pretest 
validity crosscheck with Follow-up Survey 
Item 24 


22 


To enable MEU's to use this Closure Survey ' 
to respond to current Standards 


23, 24^ • 


• J — 

Clafifying information desired by MEU*s 


25 


To calculate pretest level of client effort 


26 


To respond to*MEU*s post-employment service 
information jieeds 



2u2 . . 
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Table 22 
■Closure Survey: Sampling Dimensions 





DE 




MS 


OR 


• PA 


VA 


Final Sample 














(Goal: 300 Status 26 's 


142^ 


333 


86\ 


300 


300 


301 


100 Status 28 's 














and 30 's) 
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26"," 


100 


101 


118 


Number o'f Districts in 


3/3 


'8/42 


19/21 


32/32 


3/15 


19/61 


State Involved/Total 














Response Rate on Total 


48% 


' 80%^ . 


,66%^ 


■ 57%^ 


70%^ 


46% 



^Sample size reflects all clients closed in Statuses 26, 28, and 30 
during the sampled quarter. 

^^Instituted recommended response rate assurance procedures. 
Sample Representativeness : 

DELAWARE Sample seems to represent the fairly equal distribution of 

these three types of closures. 

MICHIGAN . The clients about whom information was collected and who 
were subsequently included in the closure survey are be- 
lieved to represent a good cross-section of all MBR Ist^ 
qtr. 26, 28, and 30 closures. 

MISSISSIPPI Statewide. 

OREGON Not applicable. Selection process inappropriate fot sample 

size requested. ; 

PENNSYLVANIA ■ Ye3 

VIRGINIA Representative of volume of clients during the same period 

of time in 1979 Regions I and II. 
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' : ' 190 ^ ' 

* . ' »■ " 

/ • * 

.' « ' * 

t ^ « , . 

• Implementing a telephone reminder 'to continued non;- respondents 
the fpll'owing week; \ , ' ' - ^ 

• Classifying unreturned surveys .as non-responses two weeks later. 

Although most states completed a few of the "questionnaires over the 
.telephone in order to check the respondents ' level of comprehension and 
"to identify problem question^ an the inistrument, Mississippi found it ' ♦ 
necessary to conduct a \arge proportion of its surveys through telephone 
and face-to-face interviews because of the primary disabling condition of 
its population (i.e., blind and visually impaired clients). This addi- 
tional effort (not immediately evident in the labor estimates provided in 
Table 17) was possible because of , the substantially smaller sample size 
in Mississijjpi. On average, the Closure Survey activity required 19 
persin days af 'PE Unit staff time to train, execute and analyze; however, 
neither clerical efforts nor clients' response time is included in these 
estimates . ^ • 

Sunmiary of Pretest Experience - ' ^ • * 

, As wjith the R-300 supplements,^ most state agencies felt that the sample 
selected for the protest of the closure survey represented a good cross- 
section of the state VR population. IVhile the usefulness of the closure 
survey uas acknowledged by most of the MEU's, the time commitment involved 
in its administration was seen by some as befing unnecessarily long. There 
was a great deal of time spent formatting, printing, distributing, collect- 
ing, and recording the data, and still the pretest process was hampered by 
high rates of non-response and missing data. To minimize non-response, the 
agencies contacted non-respondents via telephone interviews. Because of the 
problems communicating the closure survey over the telephone, the interview 
process slov^ed aown and became burdensome and^ expensive . . 

As noted earlier, the detailed reviewer comments are p'resetited in - 
another report. However, below we summarize the most frequently mentioned 
problems and suggestions noted by reviewers abqut the closure survey. 



• « • : 

- IDI 

*- 



Larger. type is needed to facilitate reading, especially by the 
. visu^ly^ii^able^. 



•'^Simplex: language should, be, Used to make the instrument .more i 
' * compatabie with the ^reading level of the -average client, - ^ 

• % . ^ • *A . * ^ ^ ' 

* • Tlie-^instiiiment'soiame and identification ijumbers should' be , 
*( ' ' \ , . 

I^lacad on the cover to facilitate handling. • , , v 

\ ■ ■ .- ■ . 

• -The questionnaire is -too long.^ ^ . v . 

' • ' ^ potential for bias exists 'due to the laqjc of response by 
certain types of cli^ents. . .• * 

• Clieryts give muKip1^re$ponses tQ questions asking for a single 
response. . ' • , > * ^ 

• Sometimes 'the availab^l^ choices are not a complete list of pos-. 
sible responses. ^' ^ 

o Use of ^standardized form with ootiorts for additional unique ^ 
state items at. the end would seem to'be an -'excellent format. 

General Data Findings from the Pretest . • ^ 

, Missing Data ^ ^ \^ 

-The amount of missing data varies considerably among the survey's 
questions and among the states. Except for a few question's having enor- 
mous differences aifiong the states, all the missing data percentages are 
stated ranges as observed across^ all the states (e»g,, *'3% to 15% missing 
data'O* The excep'feing are noted below. Missing data is defined as 
either a non-response or a response of "don 't know/don *t remember." 
IVhen the questions are compared according, to the degree of' missing data, 
three distinct clusters- emerge , Representing low, moderate, and high , 
degrees of missing data. The itenl numbers >usSd below refer to the nuiri- , 
bering scheme on the West Virginia R§T version of the Closure Survey 
(Exhibit 3). ' ^ ' , - ' / * 
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Tlje low cluster -includes (Questions 1 through 6, and' 13 through 25. 
»Thqse items inquire about the, client's current situation; employment, 
opinion of \*R serviges and counselors, and overall satisfaction Between 
3% and 13% of the data for these^ibems are missing*^ These itelatively 
. small proportions are not likely to produce analytic difficulties. 

'.The -next group cofisists.of questions 7, 8, 9, and 12, which focus 
^n changes resulting frort the rehabilitative program (changes in' ability 
t^ care for (5^s^lf, feelings'^^bout oneseljf and one 's ^future, and job- 
related skills. These items were missing between 20% ^nd 25? of the 
data. , ^ * V ' ' . • ' - ■ 

The highest proporti^On of missing dafa qccurred on items 10 andTl, 

Together, these two it'ems contain 27 si4bquest;ions on improvements in 
I , . » 

self-care^ and homemaking abilities. The data from Pennsylvania are not 
comparable to^the pther s.^ates ' data because these two items were not 
asked of Pennsylvania clients who never had problems in these areas, 
Delaware and Virginia, which used the RTC format for these questions, 
had between 31% and 39% missing data. Michigan, which asked the identical 
questions with a' different aifswer format, had missing data rates between 
23% and 26%. Thi-s suggests that Michigan's format may be the less con- 
fusing of the two. ' .> ^ t 

Moderate and hrgh*proportions (over 20%) of missing data can present 
problems for analysis. This problem is compounded by the fact that the' 
six questions in this group (items 7 through 12) all respond to data ele- 
ment 5(iv), and t>hey are the only ones on the Closure Survey to do, so. 
This issue will be addressed by 'substituting the FAI/LSI items (to be pre- 
tested) for the items included in the pretest Closure Survey, to assess 
that data element. I^urtjher, the FAI/LSI items will be asked of the client 
in peVson at closure j^^^nd' thus they need not appear in the Closu:^e Survey 
at all. 



The Michigan instrument did not allow '•I am not working'' as a legi- 
timat(i response to 'items 3 and 4 (Michigan items 2 and 3), which ask 
"hour^ worked" and "present earnings." Michigan's high rate "of missing 
data 145% and 42%, for items 3 and 4, respectively) is presumably due to 
unemployed clients leaving the question blank. 
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Time Required. of Survey Respondents ^ 

This analysis is unfortunately based on rather scanty data*. Less 
than 'a third of the survey respot^dents answered the question on how much 
time was required to complete the form/ Any, conclusions must there^^re 
^e tentative* Pennsylvania did not request 'this information on its 
Closure Survey. 

* , The data which are available indicate that, on average, it took re- 
spondents 15 to 20 minutes to complete the Closure , Survey (Table 23) . 
However, this -figure is inflated by the few respondents who took much " 
more time (up to an houx and 40 minutes) to fill out the form. Two more 
teiling statistics are that half the respondents needeji ten minutes or 
less, and 83% needed.?© minutes or less to complete the Closure Survey. 



Suggested Revisions to the Closure Survey 

In the previous s.ection of this, report, we recommended the elimination 
of data element 4(iv), and the ''satisfied' with present /situation" item 
from 8(ii). , We also recommen^d combining the counselor satisfaction items 
of 8(ii), and combirfing data elements 8(iii) and (iv) . In addition, we 
recommend th^t no questions which duplicate R-300 information should appear 
on the Closure Survey; thus/ we recommend -deleting all survey questions 
pertaining to income, sources of financial support, and closure woric 
status. Finally, the questions pertaining to functional Abilities will 
be replaced by the functional assessment inventory (FAI) items soon to 
be pretested. These items will be asked directly of the client, rather 
than through a mailback survey, and thus can be deleted. 

In effect, if our recommendations are followed, it means that the 
Closure Survey will be used solely for the purpose of addressing the 
satisfaction standard (#8) > This will greatly shorten the survey ,^^and 
should improve response rates. However, at the state's option, it may 
include questions on income , ''etc.. , if the state desi-res fo use the survey 
as a validation procedure for its R-300 data. As Well, the state, for 
its own purposes, may include the last few questions on the\^etested 
survey which a^k who completed the survey, how long it took, and whether 
further services are desired. However, all state agencies must use the 
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Table 23 

Respondent Time Required to Complete 
the Closure Survey 





AVERAGE 


NEDIAN 


NO. .OF CASES 


Delaware 


15.8 




10,1 


■ 80 


MicSlgan^ 


^ ^^14,1 ^ 




9.5 " 


269 


Missi^ssippi . 


'21.8 




15,4 


66 


Oregon ' 
' Pennsylvania^ 


. 14,7 




10,3 


193 ■ • 


Virginia 


15,8 




12,3 


180 



Time requirements not asked of clients on Pennsylvania's version 
of the Closure Survey. t 
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exact wording and order for those questions used to address Standard 8; 
any other questions must be appended to the core set of questions for 
that standard. \^ ^ , • * 

In keeping with our suggestions, we recommend that the following 
changes be made to the Closure Survey: 



delete questions T through 12; ^ 

combrine questions 13, 14 and 15 into a single question 
(satisfaction with counselor); 

retain (questions 16, 17 and 18 (satisfaction with 
service) ; 

combine questions 19 and 20 into a single question; 
allow state agencies, at their option, to include questions 
21 through 26 and other questions of interest to^ the agency 
following the core questions; 

finally) we recommend that all questions pertaining to 
satisfaction with 'services be reformatted 'to **yes-no*' 
responses rather than *Very satisfied, somewhat satisfied, 
• , • , .very ^dissatisfied/' ' * * 



Oar recpmrfiended questions for the revised Closure Survey appear 



Exhibit 4. 
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Exhibit 4 

Suggested Questions for ,the Revised Closure Surveys 



(1) Are you satisfied with your 
overall experience with the 
rehabilitation program? 
(Please check one) . i 

Yes 

No 



(2) 



Not sure or no opinion 

Are you satisfied with your 
counselor's performance , (that 
is, did he/she do a gopd job 
fdr you)? (Please check one) 

Yes^ . 

Mo 



(3) 



Not sure or no opinion 

Did your counselor arrange for 
you to have physical' restora -' 
tion^ services* such as -medical 
treatment, physical* therapy, 
artificial limbs, eyeglasses^ 
dentures, hearing aids, etc.? 
(Please check one) 

No ^ ^ ... 



I don't reftjember 

Yes 

If YES, are you satisfied with 
these services? (Pleas^e check 
dn^) 

Yes 

No 

Not sure or no opinion 



If ^ES, are you satisfied with 
the kind^of training- you re- 
ceived? (Please ^eck ome) 

Yes 



No . ' 

Not sure or'no opinion 



(5) Did your counselor help -you 
look for a job? (Mease 
check one) . • * 

Yes 1^ 

No 

Not sure or no opinion 



^If YE^, are you satlsifed 
with the help you received? 
(Please check one) 

Yes 

No ' * 

" Not sure or no 'opinion 



(6) 



Were the services oi; training 
you received from the jehabil- 
itation program useful in 
helping yol^to perform in 
your present situation or in 
helping you get it? 
(Please check one) , 



ave no opinion 

|ceived no services 
:raining from the 



i (4) 


Did your counselor arrange for 


; rel 


(abilitation program 


( 


you to have job training? 
(Please check one) 

No J 


(7) (Other) 
by Ihe 


'questions as desired 
•st?ate agency) 




I don * t remember 








Yes 
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FOLLOW-UP SURVEY i 

The Follow-up Surveys were desig^led. as a one-year post-closure mailout 
questionnaire for the purposes of documenting retention of benefits (economic 
and functional), client satisfaction, and service utility assessments. This 
information is to bemused ft)r tRe following purposes: 

• to collect data from clients^to respond to several Strandards ' 
data elementfs; ^ ^ 

• to validate follow-up information generated within the follow-up 
instrument itself; 

• ^0 compare response rates and answers at follow-up with those 

received at closure. 

An example Follow-up Survey appears in ^Exhibit 5. This Exhibit can 
be used in conjunction with Table 24, which specifies the purposes of 
each data item- on the Follow-up -Su^rvey. Note that some items were redun- 
•dant with the Closure Survey adm^inistered during the pretest. This was 
necessary to resolve the issues of (1) the optimal time to solicit certain . 
information from clients'^ and (2) the relative validity of various infoA 
mation sources. - . 

; . • - • , A 

Data Collection Methodology 

: — — ^ 

The data collection methodology stipulated two major follow-up acti- 
vities: * ^ 

• The first gathered 12-5ionth foMow-up information on 100 clients 
closed 26 during the first quarter of the preceding year, pri- 
marily to identify problems with the instrument and to get an 
estimate of one-year post-closure response rates and missing 
da^-^. / 

• The second foljl^ow-up activity was directed toward the 300 clients 
closed 26 during the pretest and on whom Supplemental C5D and 
Closure Survey informatics had been collected. This latter 
group^s responses would primarily constitute the' data necessary 
to respond to the Standards. 
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2^5 Oues!ionnatfe No I. 

VOCATIONAL REHABILITATION 'SURVEY 

i 6-MONTH FOLLOW-UP QUESlipNNAIRE 

6-7 (1) Which of Ihe following sintements best describbs your present 
situation? (Ptt\ise check only one box) 

□ 1 1^ earn a v/age or Salary at a regular job 

□ '2 I earn a wago or salary m a shcltcre(3^workshop (for. 

example, a place like a v/orkshop fo/ the blind or 
Goodwill Industries) 

□ 3 I am self-employed 

□ 4 I earn a- wage or salar'y in a state-managed Business 

Enterprise Program (that is. a state-run program 
intecyod specifically for disabled people) 

□ 5 I arrfa homernaker (that is. a' person whose primary 
^ v/orf IS taking care of„ the home) 

□ 6 'v/ork on a family farm orm a family business without pay ' 

□ 7 I am not working at present I am, A student-, 

□ 8 l«tim not working at present lam Retired 

□ 9 I ifp act v/Of king at present lam A trainee 

□ 10 OtherTP/easg cxplatny 



□ n My situation .s n6t listed above (Please explain) 



I 



8 '(2) How satisfied are you with your present situation? (Please 
chuck one) ^ , 

□ 1 y Very sjStisfiod . □ 4 Very dissatisfied 

3 f jSomev^hat satisfied □ 5. No opinion^ 

^ I □ 1 Somewhat dissatisfied 

9-10 (3) About Tow many hours.a week d© you work? (Please check one 
' . . and fill in the boxes,) 

* □ .1 usually work about I (• | hours a week. 

y □ I don't know 

O 1 am not, working. 

ll-f3 (4) What are your present earnings? (PJeasesheck one and hll m 

2^ ,^ the boxes) ^ . * 

Ic' n I earn $ 1^ — I ( per week. 

O ^ □ 1 don't knovA 

ERJC 5 * ^"^ working but 1 don't deceive a wage or salary. 
MffliffiMffitHiin U I am not working ^ • 



14-15 (5) How long have you been In your present job? (Please check one 
) and fill in the boxes) * 

□ ^ I have had this |ob about I I I months 

□ I don't remember 
O I am not working 

16-17 (6) II you usually work, have you been out of work at any tim€*durmg 
the last six (6) months?YP/GasG check one and lilfin the boxes) 

□ No: I have not been out of work at any time 

□ I don't remember 

^ □ . Yes. I was out of work fori I I weeks ^ 

□ I don't usually work * ^ ^ 

• ' * * • 

18-19 -<7) Whht Is your largest source of support? (Please check one) 

^ □ 1. Current earnings from a job. 

□ 2. Earnings from intferest, dividends, or' rent payments 

□ 3. Family and Iriends. " • V 

□ 4 Private relief agency (for example, the Salvation Army. 

Goodwill Industries 

□ 5. Supplemental Security Income (SSI) for the'^BIiHd 

□ 6. Supplemental Security Income (SSI) for the Aged * 

□ 7. Supplemental Security Income (SSI) for the Disabled 

□ 8. Aid to Families with Dependent -Children (AFDC) 

□ 9. General Assistance (GA) 

□ 10. Other public welfare (Please explain; ^ 

( 

□ 11. Public institutions (for example, a tax-supported. 

hospital, nur-sing home, treatment cenrer, etc) 

□ 12. Workers" Compensation v 

□ 13. Social Security Dwability Insurance Benefits (SSDI) 

□ 14 Veteran's* Benefits 

□ 15. Private Insurance 
* □ 16 Other: •^P/ease explain/ 

20-23 (8) How much income, if any. did you (or your dependents) receive' 
last month from ail sources of public welfare? (Please check 
"one and fill in the boxes) ^ 

□ I recelve(^ S I I., l_>_J. last month. 

□ None 

O I don't remember. » 
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24-28 . (9) How much did you earn during the last six (6) months? (P/easep 
1 check one and fill in the boxes) ^ 

' □ I earned about $ i I I I I. I during the fast 

6 months. 

□ I did not earn anything during the last 6 months. 

□ I dgn't remember. 



f 



^ 29 (10) When you finished rehabilitation services, was anyone In your 
family able to v/o^ or work longer hours at a job they already had 
because you were able to care for yourself? (Please check one) 

□ 2 No 

□ 9 I don I know 

□ 1 Yes, ' 

30 . A If yes. IS this still the case'? {Pjease check one) 

• □ 1 Yes 

□ 2 No ♦ « . 

□ 9 i^ntknow r 

' , / 

31 (11) H<^ do you feel about yourself compared to 6 months ago? 

(Please check one) 

□ 1 I feel the same about myself as I did 6 months ago 

□ 2 I feel belter aboC/t myself than I did 6 months ago" 
^ □ 3 I feel worse -about myself than I did 6 months ago 
^ 0 9 1 don t know 

S 

32 (12) How do you now feel about your future , compared to 6 months 

ago? (Please (Theck one) ^ 

□ 1 ' I feel the same about my future «s I did 6 months ago 

□ 2 I feel better about my future than I did' 6 months ago 

□ 3 I feel worse about my future than I did 6 months ago 

□ 9 I don t knovj ^ ^ 

(13) Has your ability to perf orm any of the following furjctions of self- 
care changed at all during the la^t six months? (Please check on 
the right side whether your ability to perform^ach itenj has stayed 
the same, improved or gotten worse during the last six months) 

In rhe ui-Jt 6 monins * 

n)/ <1&Htly 10 



33 a Feeding 

3^ Bathroom functions 

35 ^ c Taking medicine^' 

35 d Grooming /shaving, 

hair care., make 
, ✓ up, etc ) 

37 e Dressing and 

undressing 

38 f. Moving my body 

weight ^ 



Has stayed 


Has 


Has goiifn 


Nol » 


ihe same 


improved 


worse 


sure 










1. □ 


2 □ 


3 □ 


4^n 


1 □ 


2. □ 


3.'n 


4 □ 


1 □ 


2 □ 


3. □ 


4. □ 


h □ 


2. □ 


' 3. □ 


4. □ 


1. □ . 


2 □ 


3. □ 


4. □ 


1. □ 


2. □ 


♦ 3. □ 


4'. □ 
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39 


g. 


Moving around 
the home 


40 


h. 


Climbing stairs 


41 


i. 


' Reading 


42 


I 


Writmg 


43 


k. 


Talking 

/ 


44 


1. 


Answerinrg the 
phone 


45 


m 


Moving on the 
street 


46 


n. 


Using public 
transportation 


47 


^ 0. 


Driving a car^ 



1. □ 



^2. □ 3 □ 4. □ 



1. □ 2. □ 3 □ 4. □ 

1. □ 3D 4. □ 

1. □ 2. □ 3. d 4 

1. □ 2.JJ 3 □ ^4 □ 

1. □ 2. □ ^ 3 □ 4. □ 

1. □ 2D 3, □ 4. □ 

1. □ 2. □ 3. □ J. D 

1. □ 2i-.n " 3. □ 4. □ 



(14) If you are primarily a homemaker. has yotjr ability to do any of the 
skills and Activities related to homemakin q changed during the 
>past sjLx months? (Please check on ttie right side whether your ^ 
\ability to do each of (he iterhs has stayed the same, improved, or 
gotten worse during the last six months ) 





In Ihe last 6 
monihs my ability lo 


Has stayed 
the same 


Has 
improved 


Has gotten Not 
worse Sure 


48 


a Carina for children 
^ and family 




2. □ 


'3. □ 


4 □ 


49 


b Caring for pets 

c. CooKing and fixing 
meals 


' 1-. □ 


2. □ 


3. □ 


4. □ 


50 


• 1. □ 


2. □ 


• 3. □ 


4.,n 


51 


d. Washing dishes 


1. □ 


2-. □ 


3. □ 


4. □ 


52 


e. Cleaning house 


1. □ 


2. □ 


3. □ 


4.-n 


53, 


Doing laundry 


1. □ - 


2. □ 


3. □ 


4. □ 


54 


g. Shopping for food 
and groceries 


1. □ 


2. □ 


3. □ 


4. □ 


55 


h. Running errands 


1. □ 


2. □ 


3. □ 


4. □ 


56 


i. Budgeting and 
paying bills 


■ 1. □ 


2. □ 


3. □ 


4. □ 


57 


j. Taking part in 


1. □ 


2. □ 


3. □ ■ 


4. □ 



activities 
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58 (15) Has tKcre been any change In your ability to do j ob-related skills 
and_ activities (hke loping, using tools and machinery, assembly 
work, serving people, and so on) during th*e six months since you 
left the rehabilitation program? (Please check one) 

□ 2 No. there has been no change ^ 

□ 9 I don t know' 

□ 1 Yes. there has been a change * 

A If yes please Itst the skills <yr activities below' and 
check whether you/ have improved or gotteh worse 
m It 

m ir)Q i.i^i 6 - ^ Has stayed Has Has QOttcn 

monlns my ao»iily to ^ ^ IMe s»ime impfOved wOfSO 

— — ■ □ □ '□ ■ 

□ □ □ , 

., ^ □ □ □ 



J. 



59 (16) How satisfied are you now with the informatid^ provided by your"^ 

counselor about ^our disability and the opportunities for 
improving your situation? (Please cfieck one) 

□ 1 Very satisfied 

□ 2 Somewhat satisfied 

□ 3 Somewhat c^issatisfied ' 

□ 4 Very dissatisfied ^ 

□ 5 No opinion 

✓ 

60 (t7) How satisfied are you now with your counselor's willingness to 

listen to your ideas and suggestions when developing your 
rehabilitation plan and job goal? (Please check one) 

□ 1 , Very saiisfib^d 

□ 2 Somewhat satisfied 

□ 3 Somewhat- dissatisfied 

□ 4 Very dissatisfied 

□ 5 No opinion 

61 (18) How satisfied are you now with your counselor's promptness in 

providing services to you? (Please check one) 

□ 1 Very satisfied 

□ 2 .Somewhat satisfied 

□ 3 Somewhat dissatisfied 

□ 4. Very dissatisfied 
• □ 5 No opiniop 
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62 (19) Did your counselor arrange for you to have physical restoration 
services (such as medical treatment, physical therapy, artific^t 
limbs, eyeglasses, dentures, ^hearing aids, etc.)? (Please 
chock one) 

□ 2. No 

□ 9 I don't remember 

□ 1. Yes 



r' . ^ ' 

63 A. If yes. how satisfied are you with these services'^ 

^Please check one) 

□ 1. Very satisfied 

□ 2, Somewhat satisfied 

□ 3/ Somev/hat dissatisfied 

□ t^r^exy dissatisfied 

□ 5. No opinion 

64 (20) Did your counselor arrange for 'you to have job-training ? 

(Please cfheck one) 

□ 2. No 

□ 9. I don't remember. 

□ 1. ' Yes 

65 A. If, yes, how satisfied are you with the kind of 

training you rpceived? (Please check one) 

□ 1. Very satisfied 

□ 2. Somewhat satisfied ' 
^ □3. Somewha4. dissatisfied 

□ 4, Very dissatisfied 

□ 5, No opinion 

66 (21) /)id.your counselor help you look for a Job? (Please check one) 

□ 2. No «^ 

□ 9. I^dqn't remember. 

□ 1. Yes , .Q 

67 • A. If yes. how satisfied are you with this help'? (Please 

check one) 

□ 1. Very satisfied 

□ 2. Somewhat satisfied 

□ 3. Somewhat dissatisfied 
4. Very dis^tisfied 

□ 5. No opinion 

68 (22) How US£lui were the services or training you received from the 
rehabilitation program In helping you to get your present 
situation? (Please check one) 
D 1. The services or training were of great use. 

□ 2. The services or training were of some use. 

□ 3. The services or training were of no use at all. 

□ 4. I have .no opinion. 

□ 8. I received no services or training from the rehabilitation 

program. 

69 (23) How useful were the services or training you received from the 
rehabilitation program In helping you to perform In your present 
situation (even If they were of little or no use in helping you get 
It?) (Please check one) 

□ 1. Thd. services or training were of great use. O 1 

□ 2. The semces or training were of som^ use. ^ -I ■ 
^ / . ' □ 3. The services or training were of no use at all. 

• ' , Da. I have no cfpinion. 

:V, , □ 8. I received no services or training from the rehabilitation 
'y''^*"^'''^^^' program. ... • 
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70 (24) 



71 (25) 



72 (26) 



(27) Do you have any comments you wish to make about your case 
or the program,_or is there anything else younhmk it wouid be o 

useful for us tolcnow- z — 



74 (28) How Ic^ng did it take for you to fill out this questionnaire? 

I I I Mtnutes 

(29) Do you wish to be contacted by your vocational rehabilitation 
agency for further services? If so, please provide: 





m 




Njimfl 


e 






XHIBI 




S(a(fl AOdross 








H 














Cny 


Smifl 




-"J 


Zip 


coat * 


AfflA Codfl 


Tflltpnonfl Numbfl^ 








p- 



How satisfied are you with your oyerall experience with the 
rehabilitation program? (Please check one) 

□ 1. Very satisfied □ 4. . Very dissatisfied 

□ 2. Somewhat satisfied □ 5. No opinion 

□ 3 Somewhat dissatisfied 

Would you recommend your vocational rehabilitation agency to 
a disabled friend? (Please check one) 

□ 1. Yes □ 2. No □ 3. No opinion 

Who answered this questionnaire? (Please check one and (ill 
m the blank) 

□ 1. I, the former rehabilitation program client, answered 

this xquestionnaire by myself. ^ 
D 2. I. the former rehabllitatioxv^rogram client, answered 
■ this questionnaire with the help of another person 

This person is my: (Please give relationship) 

-. 

□ 3. Another, person answered this questionnaire for rne. 

This btlner person is my: (Please give relationship) 



Table 24 

Purposes of Follow-up Survey Data Items 



I tern (s) 


\ Purpose (s) 


1 ^ 


Pretest validity checkpoint for Follow-up 
Items 3, 4, and 9 


3, 4 > 


To respond to Standards 2(i'ii), 2(i'v) and 


5, '6 


Prete^ validity checkpoint for Follow-up 
Item 9 and Closure Item 1 

^ '■ ^ 


7 ^ 


r Tp respond to Standard 7(ii) 


8 


Pretest validity crosscheck for Follow-up 
Items 4 and 7 


9 


' i ' 

To respond to Standards 2(ii-'i), 2(iv) and 
'6(i) 


10-15 


To respond to Standard 7(iii) and as pretest^ 
validity crosscheck with'Cli'Sure Items 7, 8, 
9, lO; 11 ^ 12 ' 


"2; 
16 - 21A 

• 


To respond to Standard 8(ii) and as pretest 
validity crosscheck with Closui;e Items^2, 13, 
14, 15, 16A, 17A and ISA ' 


22 


To respond to Standard 8(iii) and as pretest 

validity crosscheck with Closure Item 19 o 

^ '* * - 


23 


To respond to Standard 8(iv) and as pretest 
validity crosscheck with Closure Item 20 


. 24 


To respond to Standard ^i) and as pretest 
validit}^ crosscheck 'tVith Closure Item 21 


25 " 


To enable MEU's to use this Follow-up Survey' . 
to respond to' current standards ^ 


26, 27, 


Clarifying information desired by MEU's 


28 


To calculate pretest level of client effort 


29 


To respond to MEU*s post-employment service 
. inforraatipn needs 
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Table 25 illustrates the sample size, proportion of district's involved 
in each state.* s efforts, and the response rate achieved for the 12-month 
and 6-month 'follow-up activities by each MEU. The range .of response rates 
was not as wide as with the Closure Survey^ spanning 37% to^% with an 
average of 51% for the 12-month survey, and from 43% to 73% for the 6- 
month survey. IVhile adherence to' the response rate assurance procedure 
did not affect the -rates as dramatically as in the Closure Survey, the 
necessity in Mississippi to conduct telephone or face-to-face interviews 
with a large proportion of the visually impaired clients did have a posi- 
tive impact on the return in that state. Despite this additional invest- 
ment, the level of effort estimates^ presented in Table 17 do not reflect 
an increased total cost to the Mississippi MEU. In fact, at approximately 
16 PE person days to train, execute and analyze the 12-month follow-up 
'survey, Mississippi actually required less than the average 19 person 
days for this activity. It should be remembered that these estimates do 
not include clerical efforts or clients' response time. 

Summary of Pretest Experience 

' 'As regards the overall process of the follow-up^ the predominant con- 
cern of the MEUs was that the adminis'tratibn of the Follow-up Survey re- 
quired i major time commitment that they felt may not have been justified 
by the results received. Even with many hours spent mailing, collecting, 
and processing the surveys, the problems on non-response were still evi- 
dent. Even the time-consuming process of telephoning yielded few contaets. 

The reactions to the Follow-up Survey itself closely parallel the 
reactions to t^e Closure Survey^in their general t^ne. Four recurring 
problems stand out: * ^ 

• larger type needed- to facilitate reading; 

.many c(uestions need to be reworded using simpler language; 

• the questionnaire's name and identification numbers should 
appear on the cover; 

• .client non-response is a potential source of bias- 
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Table 25 

12-Month and 6-Month Follow-up Survey: 'Sampling Dimensions 



• 4 


i 


• MT 

MI 


Mb 


OR . 


PA 


VA 


12 -Month Survey 
• Final Sample 
(Goal: 100 Status 26s t 


1 

100 


99 


103 


100 


102 


100 


Response Rate 


46% 


59% 


65% 


37% 


57% 


41% 


6«Month Survey 

Final Sample 

(Goal: 300 Status 26s) 


1 
( 

125 ' 


326 


■ 86 


300 


261 


301 


Response Rate 


43% 


53% 


73% 


67% 


59% 


47% 


Number of Districts in 
State Involved/Total 


3/3 • 

t 


39/43 


! 

20/21 1 


32/32 


1/15 


56/61 



Sample Representativeness: (12-Month, Survey) 

Distribution across three districts didn't represent entire 
year. 



DELAWARE 
MICHIGAN 



Mississippi 
OREGON 

PENNSYLVANIA 
VIRGINIA 



Sample not sufficiently large to make valid inferences about 
the total population of 2nd qtr. 26 closures. 

Statewide. 

Not applicable. 

Questionable, sample too smal4; larger sample needed. 
Representative of statewide volume of 26 closure status. 
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V 

A 

general Data Findings frqin the Pretest 
Missing Data 

Most of the items on the Follow-up Survey solicit essentially the 
same information as their counterparts on the Closure Survey, except, of 
coursd, that they pertain to a different time period. There are only 
three wholly new questions on the Follow-up. 

For most of the repeated items, the proportions of missing data on 
the Follow-up Survey were about the same (within 5%) as on the corres- 
ponding Closure Survey items- The exceptions were items #11 and #12 
(Closure Survey items #8 and #9) for which the missing data rate dropped 
by 8%, to 12% and 16%, respectively- In addition, the missing data rate 
'on item ^13 (Closure Survey item #10), dealing with improvements in self- 
care functions, dropped sharply from the 31%-39% range on the Closure 
Survey (for states using the West Virginia format) to 16%-23% on the 
Follow-up- This may, be due to the simpler answer format used on the 
Follow-up. In the earlier discussion of the Closure Survey, it was noted 
that Michigan's lower missing data rate for this question was probably due 
to the clearer* answer format on the Michigan instrument- The Follow-up 
Survey data tend to support this conclusio.il- 

Three new questions were added to the Follow-up Survey- Item #5^ asked 
how long thfe client had been in his present job, and item #6 asked if he 
tiad been -out of work at all in the previous six moriths- The relatively low 
proportions of missing data for items #5 and #6 (11% 'and 12%, respectively) 
suggest that clients did not have much trouble answering these questions. 
This was not the case with the third new question (#9) which asked about . 
earnings for th^ previous six month period- One fourth of th^ clients did 
not answer this question, perhaps because they could not remember- - 

Time Required of Survey Respondents 

Table 26 presents the average. and median times required by respondents 
to complete the Follow-up Survey- "The average time ranged from 12.1 min- 
utes to 20.5 minutes- The high figure found for Mississippi may result 
from the fact that the ^surveys were administered face-to-face, thus 
♦'forcing" clients to answer the maximum number of questions. As well. 
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. ' Table 26 
Respondent Time Required to Complete 
the Fol low-Up Survey 





" AVERAGE 


MEDIAN 


NO. OF CASES 


Delaware 


■J, 

12.1 


10.4 


51 


Michigan 


12.8 


10.3 


51 


Mississippi 


20.5 


15.5 ^ 


55 


Oregon 


18.5 


14.8 


36 


Pennsylvania 


12.3 


lo.o 


43 


Virginia 


12.5 


10.3 


48 
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*'in face-to-face intelrviews clients have an opportunity to request clari- 
fication of unclear *quest ions. To the extent such clarifications were 
requested, time requirements would increase. 

The minimum time required was two minutes; obviously, a client 
who declined to an'^wer most questions. The maximum was 90 minutes. 

Suggested Revisions to the Follow-up Survey 

Many problems with the administration of the Follow-up Survey might 
be linked to probl.ems with the instrument. Improving the instrument may 
'lead to a faster, higher response rate, thus eliminating some of the above 
concerns. At the beginning of the pretelt, the basic purposes of the 
Follow-up Survey were twofold: 

• as a primary source for data elements 2(iii)., 2(iv), 6(i), 
7(i), 7(ii), and 7(iii)rand 

• as a 'back-up source for data elements in Standard 8. 

As a back-up source for Standard 8, the Follow-up Survey is conceptu- 
ally less reliable, less valid, and in practice produced a higher level 
of missing cfeta than the Closure Survey. We recommend, therefore, elimi- 
nating alj tjuestions* pertaining to client satisfaction from the Follow- 
up Survey. 

We h^ve recommended that, data element 6(i) be eliminated ^rom the 
Performance Standards. Further, for data element .7 (iii) , dealing with 
functional abilities, we have recommended substituting the Functional . 
Assessment Inventory (FAI) and Life Status Indicator (LSI) items selected 
after pretesting. Data collection will be deferred until after the pre- 
test. Also, regardless of the specific items chosen,,' those items will 
have to be converted into a format suitable for self-administration via 
a mail -back survey. Finally, in order to implement ouy recommendation 
for data element 7(ii) -- which requires a more detailed accounting.-of . 
financial sources of support we will need to add a question hn non-wage, 
private sources of income. 

As with the Closure Purvey, state agencies may 'add other questions' 
to the Follow-up as they wish. However, all states must use the exact 
wording and order for the questions used to address the standards; 
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additional questions must be appended to the core set usecl for the 
standards. 

Bfsed on our suggestions, we recommend that the Follow-up Suarvey 
be changed as follows: 

• retain question 1 (current employment situation) to "set 
the* stage" for answering the survey; 

• delete questions 2 and 3; 

• retain question 4 (weekly earnings); 

• delete questions 5, 6 and 7; 

• retain question 8 (income from welfare); 

• delete question 9; 

• substitute the FAI/LSI items — after they are pretested 
and converted to a- self-administering format for 
que'stions 10 through 15; 

• delete questions 16 through 24; 

• allow state agencies, at their option, to include questions 
25 through 29, and other questions of interest to the agency 
following the core questions; ' ' 

• add a guestiojn on non-wage, private sources of income- . 

Our f ©Commended questions for the revised Follow-up Survey appear 
in Exhibit 6» . . 
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^ Exhibit 6 
s Suggested Questions for Revised Follow-up Survey 



(1) Which of the following state- 
ments best describes your 
present situation?' (Please 
check only one ) 

I earn a wage or salary, 

either at a regular job or 
from self-employment 

I earn a wage or salary in 

a sheltered workshop or BEP 

I am a homemaker > 

I work in a family farm or 
business without pay 

I am not working at present 

Other (explain) 



(2) How much income^ if any, did 
you or your dependents receive 
last month from all sources of 
public welfare? (Please check 
one and fill in the space) 

We received $ 

last month 

None 

I don't remember 

(3) iVhat are your total earnings 
^from a job, self-employment, 
sheltered ^vorkshop, or BEP)? 

^^^^ I earned J >_ last 

week 

I am working but I don't ^ 

receive a wage or salary 

I am not working 

I don't know' 



(4) IVhat'was your income last 
month from private sources 
other than the earnings re- 
ported in Question 3? (For 
example, from rents, divi- 
dends, or private insurance.) 

I received $ 

last month ^ 

None 

I don't remember 

(5) (Functional Assessment Inven- 
tory (FAI) and Life Status 
Indicators (LSI) items selected 
after pretesting, and reformat- 
ted for self-administration) 

(6) (Other questions as desired 
by the state* agency) 
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VI. INVESTIGATING PROBLEMATIC PERFOTIMANCE : 



SUPPORTIVE EVALUATION USING A DATA-BASED 



DECISION SUPPORT SYSTEM 



In the preceding two chapters we 'discussed the Performance Standards, 
their associated data elements, and the data collection instruments needed 
for their implementation. These standards , 'along with the Procedural^ 
Standards discussed in later sections, comprise the - descriptive component 
of the overall standards system. From the descriptive analyses come simple 
assessments of states' performance in relation to expected performance 
levels. This information will provide an overall picture of state and 
program perforipance; but, by itself, such information provides only part of 
the data neecls-^of the standards system. The primary intent of the standards 
system is to facilitate improvement in state' performance. However, the 
descriptive data on state performance will not, in general, provide any 
clues as to why stages exhibit less-than-adequate performance, or what 
acljions might Ue taken to improve performance. Thus derives the neeci for 
a second, analytical aspect of the standards system. 

We expect that the standards r^orting system will provide clear 
indication that some agencies on some data elements will not have met 
their objectives for level of attainment. The standards system does not 
stop there, however, but instead moves to analyze causes for the problem- 
atic attainment and to develop corrective actions, using the supportive 
evaluation system . .The main use of >this sy;5tem is to identify things the 
various actors in the VR system could do differently to improve the attain- 
ment of tlfe state VR agencies. The putpose of the supportive evaluation 
system is to close the gap between reporting on the standards and actions 
based on the standards. The supportive evaluation system shojjld: 



ensure that standards have a positive impact on state and 
program performance; 
• provide an ability to pinpoint causes for problems in sub 
units ' performance ; 
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• identify strategies leading to enhanced performance; and 

• identify appropriate policy recommendations and program 
actions which can ]fe taken by state agencies, RSA, or 
Congress, based on the 'analysis and aimed at improvement 
in state performance. > 

Achievement of these objectives will require synthesis of analytic 
research techniques, first-hand familiarity with program operations, and 
a sensitivity to policy concerns. Sensitivity to policy concenis is per- 
haps the most Important consideration in terms of the overall design of 
the supportive evaluation model. Decisions are. made by program, managers, 
be they within RSA or within state agencies. Supportive evaluation is 
intended to inform decisions aimed at alleviating observed problems, in 
agency performance. As such, the overall supportive evaluation^ model must 
first and foremost address the information needs of program managers. 
What this means in practical terms is that, at a minimum, program managers 
must be provided with information that is; 

• relevant to the issues (i.e., pro{)lem) under consideration; 

• quickly and easily interpretable; ^ ' ^ 
timely; and 

• suggestive either of an immediate policy response to the 
problem, or of further investigation needed before an 
appropriate response can be formulated. 

With design of such a system, program managers are provided with a 
tool for using program data to move from problem identification, to analy- 
sis of causes for problem performance, to programmatic actions that are 
based olh the program data and are aimed at alleviating the problem perfor- 
mance. Thus, the supportive evaluation design comprises a data-based, 
decision support system . ^.^ 

GEN'ERAL PROCESS FOR CONDUCTING SUPPORTIVE EVALUATION 

- The basic flow of the supportive evaluation system is shown in Figure 
5. Problematic attainment, \N4iere an agency is unable to meet its agreed- 
on objective for a particular standard data element is the signal for the 
process to start. First, program managers, within RSA and within the 
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Figure 3 



The Flow of the Supportive Evaluation System 



state> VR agency 
operations 




yes 






no 


program managers 


investigate 









yes 



Implement corrective 
actions 




evaluation 
research 
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state VR^ agencies, investigate the problematic attainment f If they are 
able to identify problems and possible corrective actions, then implemen- 
tation is the next step.. If not, then* more formal evaluation research is 
called for. Implementation of the corrective actions will affect state 
VR agency operations in the next cycle of the standards system. As a 
result of the corrective actions the agency may be able to meet its objec- 
tives. Otherwise, the cycle starts anew. 

As noted, the investigation of problematic attainment has been broken 
into two parts: 

^ basic problem identification , . carried out by program man age j:s 
within RSA and within the state VR agencies, using the standards 
and MIS reporting systems plus the managers' knowledge of program 
operations, and resulting in imifiediateV corrective action; and 

• evaluation research , carried out by evals^'tion researchers within 
JISA or within the state VR agencies, or by outside consultants, 
using the proposed MIS and other data bases as well as requiring 
primary data collection*, and requiring some time lag before cor-, 
rective action can be formulated and implemented. 

These two parts differ then in who carries them, out, but especially to 
the extent tha^ the basic problem identification occurs in a timely 
fashion, using the reporting system and the MIS. If evaluation research 
is required, then most likely corrective actions will not be possible in 
time for the next cycle of the process. In fact, the results of the 
evaluation research may not be available for a year or more, given the 
nature of evaluation research.. This lag is the reason that the investiga- 
tion of problematic attainment is brokert^into two parts, so that timely ^ 
corrective actions can be taken, if possible. However, the division between 
these two types of investigation is not absolute --Inhere is some overlap. 
This chapter focuses on the basic pro(3^ess of problem investigation and is 
organized as follows: 

• a gerferal model of the thinking process that managers would 
undertake to investigate the causes of problematic attainment 
is presen^ted first; anJ 

• two examples of how the thinking process would be applied to data 
on the revised standards are given next. ^ 
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THE PROCESS OF PROBLEM iftENTIFICATION 



The profcess of problem identification outlined below is to be carried 
out by program managers, within RSA and within state VR agencies. The' 
information for the problem identification will come from the standards 
reporting s/stem, as well as from the managers' knowledge of program 
operations. The process consists of tracing the possible problems by 
first organizing the components of the standard, then examining as second-^ 
level indicators other standards' data elements and other' informational 
elements of the reporting system. Examination of these will then lead to 
further examination of third-level indicators, and so on. At any point in 
tracing out these indicators the problem may be identified to the manager's 
satisfaction. At that point, corrective action is formulated. Or, at any 
point in tracing out these problems, further ^analysis in tlie form of eval- 
uation research may be required. This process can be shown as a tree, not 
unlike a decision tree, as in Figure 4. Of course, the process of problem 
identification may lead down severkl paths at once. The point is to do 
the analytical thinking and utilize existing information to identify pos- 
sible problems and corrective actions^. Also, more than two pathfl may need 
investigation from a particular node, and more than three levels of indica- 
tors ma,v have to be examined. Below, the specifics of this process are 
further dclineafed. 

If a data element shows problematic attainment, the. first level of 
analysis is to examine the components of the element, dissecting the ratio 
or measure into its separate parts, to pinpoint the areas needing attention. 
For example, if the numerical value of a ratio is too large, the problem 
may be in the numerator (too large), the denominator (too small), or both. 
Comparison of performance on the data elements .or their components with 
that of other agencies with similar programs, or historically, or on other 
data items, can help determine the extent to which the indicator shows a 
real problem, or if there i^ a good explanation for the performance. . The 
goal in this analysis is to seek explanation , or the identification of 
which cumoonents or related measurej^ pinpoint the areas to be explored 
further. Tliis analytical process may take several iterations before a 
cause is pinpointed. The first levels of the process are not to be seen 
as complex statistical- analysis problems but, rather, straightforward,, 
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Figure 4 ^ 
Process of Problem Identification 
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% "informationarelement" is a piece of data that 'comes from the MIS or other.rpparting system, but 
is not a standards data element. ■ . . , ; . f 
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simple program comparisons which allow people to progress through a 
decision- tree, diagnosing problems and using program information to reach 
conclusions about probable causes. Some branches of a decision-tree pro- 
cess may lead to problems or investigations fl&iich require complex statis- 
tical analyses, but only several levels into t^ie process. 

Table 27 shows the decision-steps in an example exploration; this is 
a model for investigating the possible causes or problems if "expenditures 
per 26 closure" (Data Element 2ii) is problematic. 

The columns showing "first level indicators" shows four possible 
combinations of two other indicators, cost/closure and cost/case, which 
we suggest using in conjunction with an unacceptable (high) value of data.^ 
element 2ii. Depending on acceptable or unacceptalple levels of these 
indicators, a different "scenario," or type of problem, is identified. 
For instance, if both of these indicators are "acceptable," then this 
indicates that the agency is achieving a proportion of 26 closures which 
is too low. This can be confirmed by referring to Standard Data Element 3i. 
If cost/closure is unacceptable, but the cost/case-r is acceptable, then the 
agency is achieving too few closures. As can be seen here, this first level 
diagnosis leads to in-depth investigation of different parts of the system. 
The table shows the types of second- and third- level questions that could 
be pursued, depending on the initial comparisons and explanation. 

At each level of the investigation, the goal should be to quickly and 
more finely hone in on the precise nature (i.e., cause) of the problem. 
Depending on the findings generated by a given level of the analysis, the 

4 

program manager may decide either: that further investigation is warranted 
before' formulating a policy response; that the findings are adequate to 
suggest an appropriate response; or that, despite the adequacy of the find- 
ings, no useful policy response can, be offered (e.'g., due to prior institu- 
tional, legislative, or funding constraints). In the discussion which 
follows, we illustrate the chronology of thinking an4 analysis w.hich should 
occur prior to the formulation of a policy response, using numerical example 
from the MEU data collected for, the pretest* or available from reports. 
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Table 27 

Investip.atinfi Inndc*quutc Perfgrmancc on Data Element 2(iij^ ; 
f:xpcndittireh Per 26 Closure. 



Sceft 
aricr 



I-irst Level 



Cost/(,losurc 



Accept ;iblc 



I nJ » on tors 
t7Cuse 



Acceptable 



Impl i « 
cations of 
rirst level 
Indicators 



At;cncy is 
achieving 
too low a 
proportion 
of .26 clo- 
sures 



.Standards Data 
Llement 3(i) j 

f'J^ ) 

\^#26+»28*#30/ 



Second Level 
Indicators 



"Leading 
Questions'* 
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Is tlie % too 
low? 
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Ihird Level 
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column) 
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• Service^ costs 
to: 
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28s and 30s 
08s 

(Abt fUS E4) 



• Service costs 
by service 
type 



Research Questions 



(Conduct Outcomes Analysis 



1 . IVhat proportion of total costs go to ad- 
mini'stration? 



2, )Vhat is the average Jifc-of-case cost for 
each closure group? 

3. What proportion of total life-of-case • 
costs are spent on each closure group? 



1. What proportion of current service co^ts 
went to each service type? 

2. What is the average cost of each service 
type, for clients receiving that service? 



Un-. 
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cl ients 
too 
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/<>264f>28>*30 ^ 
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Un- 
acceptable 



Agency has 
recently 
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neck in 
i ntake 
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too few 
cl ients 
being 
accepted 
into the 
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1 . I^ the 
service 
process 
too slow? 



1. Timeliness 
10-12/12-24 
R-300 item 3,M,2 
Average time from 
acceptance to 
closure (10-24) 



Which aspect of services for accepted clients 
takes relatively too longf 



2. Have we 
had a recent 
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acceptances?" 



Abt MIS element 
1:9: rate of 
acceptance 



None (end of investigation) 
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Element 1 (ii) 
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1 . Do we have 
too few 
applicants? 



if of applicants 
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E9: Rate of Accept- 
ance y ^ 
0 of new status I0s\ 
* new applicants + ' 
^ on-hand applicants) 
*c^on-hand t)6s / 



2. Does use 
of Extended 
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account for 
the low 
acceptance 
rate? 
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TlPOe- (06 takes 
too^long) . 



2. Abt E9c: 
02 => 06 
(too many enter 
06) 



1. What kinds of clients are going into 06? 
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06? 
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applicants? 
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1. What reasons are given for closing clients 
ineligible? ^ \ ^ 
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referred? . * 
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EXAMPLES OF THE 'PROCESS OF PROBLEM IDENTIFICATION 



The discussion presents two working examples of the supportive evalua- 
tion process, '^'hese two examples include investigating a program-related 
data element (Standard '2(ii), cost per 26«closure, which was used also for 
Table 27) and a plient-related data element (Standard 5{i), percent 2^ 
closures competitively employed)* These analyses will include indicators 
from the Management Information System (MIS) standards ' data elements, 
and information contained on the R-300, RSA-2, and RSA-101. Clearly, 
timely ahalysis will -rely on the availability of such reports without exten 
sive delay. 

As will be shown, the^indicators used i^ the investigation of problem 
performance are grouped and sequenced in such a 'way as to answer increas- 
ingly detailed qu^tions. This allow? managers to go a fair distance in 
determining the* nature of the problem before needing recourse to more 
sophisticated and time-consuming '^causal" analyses (that is, evaluation 
research). This is not to say that more sophisticated Analyses are'un- 
desirable or unnecessary. On the contrary, they as often as not may prove 
useful to managers in pinpointing precise causes of problem performance. 
However, the ^vantage of this model is that it allows managers to quickly 
investigate and discard certain hypotheses regarding the problem's cause, 
and therefore to more quickly direct the investigation toward what .seems 
to be the likely cause- ' Onte the likely cause is identified through use 
of the indicators, the manager can direct his evaluation/research staff to 
conduct the needed causal analyses. * ' . ' ^ - 

One important point must be made before this presentation of the 
investigation process can begin%^In our ^examples , we will use actual 
results ^pom the MEUs obtained in the pretest of the Performance Standards. 
The purpose is not. to judge agency performance or attempt to alter current 
practice, but rather to offer a more concrete example of the use of the 
system. The conclusions we present only reflect the amount of information 
available to us, and should be interpreted only to the extent that they 
follajv from the results prel^nted. MoreoVer, there were no objective^ .set 
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for these agencies. The attainment of an agency on a data element really 
has no meaning without a comparison to what was expected of an agency. 
That is, a low value on a data element might or might not indicate problem 
atic attainment , with this in mind, let us turn to a specific example 
using data element 2(ii), costs per' 26 closure.. But, remember that the 
analysis is hypotheticar . 

An Agency-Level Data Blement ' 
The Problem 

In investigating thfe performance of pretest agencies on Standard 2(ii' 
shown in Table 28, the program manager finds the following performance for 
these three states: 

Table 28 



Cost 


Per 


26 Closure 




Delaware 




$.4,461 


30 


Pennsylvania 




• $ 2,408 


64 


Virginia 




$ 3,892 


78 



From this table, we see that one state, Delaware, .has an unusually 
high cost per 26 closure. Pennsylvania and Virginia are located in the 
same region, and are used for comparison' purposes . 

Fifst Level Indicators 

r 

In order to analyze performance on this data element, the program 
manager must attempt first to determine which of the data element's com- 
ponents explains **the problem": specifically; has the agency failed to 
Mtain'd lar^e enough number of 26 closures (i.e., the denominator is 



In an actual state investigation, the manager could use additional 
information such as our comparisons to past agency performance, other 
similar agencies, performance expectations or other baseline data in 
addition to relying only on the Standards' performance level. In our 
analysis, we rely solely on the pretest data. 
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sma^l relative to the numerator)? Or, alternatively, does the agency 
have a problem controlling its various costs (numerator large relative . ^ 
to denominator)? It may appear that, in the context of this data ele- 
ment, the two issues are interrelated and inseparable: if costs per 26 
closure are high, then by definition the agency has l?oth "lost control" 
of its costs relative to the number of 26s produced, and it has failed to 
produce -an appropriate number of 26s given its expenditure level. However, 
despite the intuitive sense of this linkage, we feel that the numbers and. 
cost questions can and need to be separated in analyzing the data element. 
This will be shown below. 

The method developed for analyzing this data element is as follows : 
First, we make a preliminary assumption that the problem lies in the 
agency's " 26 factor" ; that is, "in the speed and frequency with which the" 
agency produces 26 closures. Only if the first and second level indicators 
suggest no production (26s) problem, will the manager undertake a cost 
analysis. , 

Thus, the first quest ipji th.e manager asks is "why do we have so few 
26 closures?" One way to analyze this question is to place it in the con- 
text of a client flow problem; ^That is, the agency's low number of 26 
closures may be the result of some bottleneck or failj^re in the service 
process.' This is the approach taken below, which identifies four separate 
flow problems (moving from the. latest to'the earliest phases of the service 
process) : " . 

1. The ag^cy is rehabilitating too small a proportion of 
its acciej)ted closures. 

2. ' The ag^^cy is achieving too few closures, in general, ^ue 

to bottlenecks in. the service process for accepted clients 
(timeli?^06S problem). Stated from a different focus, the 
accept^ clients of the sub-unit are spending relatively 
longer periods of tilSe in the various service statuses. 
This reduces the pool of ^clients nearing closure,, and thus 
reduces the pool of potential 26 closures. 

3. The agency is accepting too few clients, thus 'cutting 
off the flow of potential 26 closures (intake problem) . 

4. The agency has both an intake and. a timeliness problem. . 

... 239 '* ■ ■ , 
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The manager can test each of these hypotheses hy looking simultane- 
ously at two related indicators: total costs per closure; and total costs 
per accepted case. These two indicators use the same numerator (total 
'expenditures) as is used in data element" 2 (ii) , However, they "spread'* 
the expenditures over larger groups, and by analyzing the size of those 
larger groi;q)s in relation to expenditures, the manager gets an idea of 
wh^t (if any) kind of flow problem exists. 

The results of this investigation follow: ' * 
- ' Table 29 . 

Cost Per Closure and Cost Per Case 



J 





Total Cost/Closure 


Total Cost/Case ■ 


Delaware 

Pennsylvania 

Virginia 


$3,428.60 
1,757,51 
2,429.91 


- $1,573.69 • 
714.58 
1,108.10 



(AvailabJe on the RSA-2 and RSA-101) 

• Based on Delawar^^ comparison to the other two agencies, the program 
manager concludes that Delaware has both an unacceptably high cost per 
( closure and cost per case. The former suggests that clients are moving 
through the service statuses at a slow pace (a service bottleneck, due to 
a timeliness problem or to an increased provision of long term services) • 
The latter suggests the existence of an intake bottleneck. ' , 

Second Level Indicator for Investigating High Cost Per Closure 

As a, rough test of the existence of servij:e bottlenecks (as evidenced 
by a high cost/closnre) , the program manager turns 1:o the second level 
indicator, Post-Acceptance Closure Rate (MIS element Ell). This indicator 
compares the number of cases closed during the reporting period to the 
number of open cases. If this ratio declined over time it would indicate 
a slowing trend *in the flow of closures. For'any given reporting period, 
a low ratio.-- "low," that is, in relation to other baseline figures '-- > 
indicates a service flow which is "too slow," and^ perhaps in need of i 
adjustment^. ^ 
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The results of MIS element Ell follow: 

Table 30' . 
Posc-Acceptance Closure Rate 





Delaware 


.847 * 




Pennsylvania 


.685 




Virginia 


.838 



(Available on the RSA-101) - Jl 

Based on Delaware's comparison to the two othei^. agencies, and in the 
absence of other baseline .information, the Post Acceptance Closure Rate 
is found to be within acceptable bounds. This rejects the service bottle- 
neck hypothesis. The manager would then examine past data 'on the number 
of Spplicants desiring VR services to see if a recent influx of applicants 
is skewing^e results to make it appear that the cost/closure is problem- 
atic. (We domJ^sJia^ the historical data to make this examin^ion.) If 
this does not explain the high cost per closure, then the program manager 
would assume it tb be a cost problem and investigate from that angle-. 

Second Level Indicator for Investigating High Cost Per Case 

The next situation to be examined is Delaware's high cost per case. 
It -is possible that the high cost per 26 closure may be the result of an 
intake bottleneck that is, the number Qf the closures is low because the 
agency is accepting too few clients into service, thus cutting off the flow 
of potential 26 closures. As the first test of this hypothesis, the man- 
ager looks at the agency's rate of acceptance MIS Element E9r The 
results of this investigation follow: , 

^ As the first test of this hypothesis, tjie manager looks at the agency' 
rate of, acceptance MIS Element E9. The results^of tbis investigation 
follow: . * . ^ -r 



: ' . Table 31 

r A Rate of Acceptance 



• ? - 


Delaware 


.389 




Pennsylvania 






Virginia 


.435 1 " 
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Before conchiding that Delaware has an^ intake bottleneck, the prograi 
manager will also want to examine the accepted VR population relative to 
the state's population (nijfnber served per 100,000 population. Standard 1, 
data element iij. This investigation reveals the following: 

Table 32 

Number Served Per 100,000 Population 



Delaware 


404,6' 


Pennsylvania 


690,6 


Virginia 


474,1 



Based on a low rate of acceptance and a low number served per lOO.^^OOO 
population, the program manager concludes the existence of an intake bottl 
neck, and turns toward an investigation of the intake process. « 



Third Level Indicators and "Leading Questions" 

There are three possible explanations of this intake problem. They 



are: 



* • Are there too few applicants? 

• - Is "there an overuse of the extended evaluation status? 
^ Are there too many ineligibre applicants? 

To examine the role the use of- extended* evaluation played in the low 
acceptance rate, the program" manager must, answer two questions: 

• .Were too many clients placed into extended evaluation? 

• Did they stay too long? 



To answer the first question, the manager examines the percentage of 
.clients placed into extended evaluation. The results are: 

Table 33^ ' , 

Percent of Clients Placed in 06 



Delaware 


16% 


Pennsylvania 


1.2% 


Virginia 


6.5% .y^' 



(Available on the RSA-101) 
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iVhile a determination that status 06 has ,been overused would also be 
based on the historical use of status 06 in Delaware and the types of dis- 
abilities common to 06's, the results seem to indicate an unusually high 
percentage of .clients jplaced in extended evaluation. (This might be inves- 
tigated further by obtaining data on the average time in status for 06 
.clients:) Suffice to say that the available figures. seem to indicate an 
overuse o^ status 06. However, the knowledge of program may indicate that 
this^ is not the case, or that evaluation research is required to answer 
the issue* 

In addition to the findings concerning the use of extended evaluation, 
the manager will also want to look at the percentage, of ineligible appli- 
cants. This investigation reveals: 

Table 34 

• ' Percent Closed 03 







Delaware 


39 


3% 






Pennsylvania 


31 


8% 






Virginia 


41 


6% 



(Available on the RSA-101) 
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Based on these figures,' the percentage of clients declared ineligible 
for services does not appear to be a source of the intake problem. 

The last area to^be investigated, in th^ effort %o locate the source of 
the intake bottleneck is rhe number of applicants for.VR services. This 
information is heavily, dependent on historical information which is unavail- 
able to us, but for the purposes of example we will use the state population 
as a guideline to allow for an across-states comparison. 

Table 35 ^ ^ ' ' , 

Applicants as a Proportion of Population , . 





i 

Applicants 


State 
■ Population 


Applicants „ _^oo 
State Population 


Delaware 


2,510 


582, OOO" 


.43% 


.Pennsylvania 


62,627 


11,731,000 


.53 V 


Virginia 


23,419" 


5,197,000 


.45% 



(Available on the RSA-101 and Census Projections) 
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" Based oh these results, the, following conclusions can be dvaim cpncem- 
ing the intake process in Delaware; 

• The number of applicants could be a source of theproblem, 
depending on the numbers of appliqants. in past years. If 
this turns out to be a problem, then the program managei* 
will want to examine putreach methocls to see if they ckn 
be made more' effective. 

• The extended evaluation status seems to be overused, thus * 
-.^reducing the overall acceptance rate. , Average time fn this 

status also needs to be investigated*. If this turns out to 
^ a problem, then the manager will want to examine the 
types of clients that are being >pta^&e^-rtfrS)6 , as well as 
the*kinds of services that are^eing plannec 

• The number of clients declAved ineligible for services does 
' not appear to be a souT^ of the intake problem. 

A ClientTltfvel Data Element ^ 

In this discussion, we present another example of the process to-be 
used by program managers in investigating problem per^boriance, this time 
using a client-level data element instead , of aftr agen^^^vcl one. 
(Again, we should remind the reader that the results shown are taken from 
our pretest results, but we do not attempt to /make firfn' conclusions con- 
cerning agency performance in the absence of other' important information. 
Use of these results is intended solely for example.)' 

T\\ e Problem ; , 

In the results of Standard 5(1), the program manager finds that three 
states show unusually low performance. These states, are Virginia,, with 
74% of its 26 ^closures being comj^etitively employed; P^nsylvania, with ' 
75. 3"^; and Mississippi, with 42. 4^^. The determination "that these readings 
may be problematic is based on comparison with past performance, other 
similar sub-units of the program, performance expectations, and other base- 
line data (the results of the three states to be investigated here are 
simply the lowest three of six pretest states. They do not reflect the 
above considerations, since that information -i^ not- available.) 
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First Level Indicators 

The potential problem identified is thatt a significantly large pro- 
portion of 26 closures are not being placed into competitive employment. 
With this in mind, the first question asked by the projram manager is 
*UVhat happened to those non-competitively employed 26 closures?** To 
^answer this^ the program manager goes to the R-300*s for 26 closures and' 
selects for work status at closure item 31) . 

This investigation reveals the following information: 
. Table 36 , 

Non-competitive Closures ' , 





Virgitiia 


Pennsylvania 


Mississippi 


Sheltered Worksliops 


11.3% 


.7% 


11.1% ► 


Business Enterprise 
Program. 


r 




2.2^ 


Homemakers 


13.6% 


26% 


43 . 3% 


Unpaid Family -Workers 


1.1% 




1.1% 


TOTAL 


.26% 


26.7% 


' 57^^% ^ 



I 

(Available on the R-300) 



.Of course, this information , cannot be interpreted independently; it . 
needs a qualifier to put it in the necessary context. Once the program 
manager knows where the non-competitively employed 26 closures went, the 
next question to be answered ^^*Are these non-competitive closures 
appropriate?'* More important , **Is this what the clients wanted?** To 
answer this, the program manager again goes to the R-300^s for 26 closures 
and gets vocational goal vs. 'outcome information (Standard 5(iii)). 
The findings from this investigation follow: 
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Table 37 
Vocational Goals a&id Outcomes 





Virginia 


1 

i .Pennsylvania 


1 

Mississippi 


Competitive^^ Gpal 


80, S%' ' 


76,0% 


45.5% 


Non- Competitive Goal 


, 19.5% 


24.0% 


54.5% 


Competitive Outcome ^ 


74.4% . 


73.3% 


42% 


Non- Competitive Outcome^ 


'25.6% 


26.7% , 


■ 58% 


Comfpetitive Goal - 
Competitive OutcoiAe 


71% ^ 


73.0% , 


.38.6% . 


Competitive Goal - 
Non-Competitive Outcome 


9.5% 


3,0%' 


6.8% 


Non- Coiiipetitive Goal - 
Competitive Outcome 


3.4% 


:3% 


3.4% 


Non -Competitive Goal - 
Non- Competitive Outcon^e 


16.0% 


23 .7% 


51. i% 



With this information being used in conjunction with the work status 
at closure (see previous table), the program manager .is now in a position 
to make some conclusions about agency performance. 

Virginia ' ^ ^ 

On the surface, the '*^under-achievemgnt" 'factor (contpetitive goal and 



non-competitive outcome) seems to be very (9 * 5%) . -Sut also noti^ce 

that Virginia has the highest percentage of clients desiring competitive 
employment as their ^vocational goal* In the^abseiice of information of 
past performance (and perform^ces *of other states) , Virginia seems to 
have ^encouraged niany of their clients to strive for competitive employment. 
UTiile a large percentage of clients desiring, competitive employment did 
not achieve this (9.5%), Virginia* seems to have made an effort to maximi^ze 
their rate of competitively employed 26 closures. Of those not^ competi- ^ 
tiv^ly employed, they are almost evenly distributed between sheltered work- 
shop employees and homemakers. This would indicate that there is no 
**creaming** o^ clients into the homemaker status. Based on these consider-* 
ations, Virgania'^s performaiT^ij^on this data element does not appear to be 
oroblvoma.tic . 
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Pennsylvania' : T 

MPennsylvania shows a high correlation between vocational goal and 
outcome. The ''over- achievers" (clients with non-competitive goals and ^ ,^ 
competitive outcomes) and ''under-achievers*' (competitive goals and non- !^ ^ 
competitive outcomes) have been minimized. Upon seeing the high correla- 
tion between goal 4nd outcome, the program manager would want to examine 
thi^ for other similar states as weLl. The manager might find that few 
states have such a high goal-outcome^correlationship, since it is per- 
fectly acceptable for client's and counselor's views on an appropriate 
vocational goal to change as the program progresses. If this investigation 
reveals a potential problem, the manager might want to go to another step 
and examine the cost per closure (available on RSA-101 and RSA-2) to see 
if the effort is being made to persuade clients to strive for a higher goal . 
This potential lack of effort is reflected in another 'area that may be • 
problematic, the use of the homemaker status. Here we find that practically 
all of the non-competitive closures were closed as homemakers (26% home- 
makers out of 26.7% non-competitiveZy employed). As such, som6 (hypotheti- 
cal) recommendations for action are possible: 

1. Persuade counselors and clients to strive for a higher closure 
status than the goal mandates. Instill in them that changes 
in vocational goals are natural and not automatically undesir- 
able. ^ * ' • ' 
^ 2. Tell counselors, possibly through a directive, not to overuse 

the homemaker status. Counselors must attempt to use the other 
non-competitive statuses, or try for competitive employment. 

Mississippi 

Mississippi shows the highest percentage of clients desiring non- 
competitive employment. Over 50% of Mississippi's 26 closures desired, 
and achieved, non-competitive employment. Mpst of these were homemakers 
(43.3%), or employees in sheltered workshops (11.1%). But for a blind 
agency, is a figure of 42% appropriate for competitive employment, 6r to6 low? 
(At this point, the program manager would want to compare the percentage 
0/ competitively employed 26' closures in Mississippi to another similar 
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, blind agency,. In our pretest, there were no other blind agencies. But, 
for the purposes of this exercise, let us assume that 42% is too low and 
that the result could be improved. Also, exogenous factors in Mississippi 
might influence the possibility for competitive employment. All these 
factors would influence an actual investigation.) Hypothetical recommen- 
dations for action follow? 

1- Tell counselors to try to get clients to strive for compet- 
itive employment. Counselors must try to convince blind 
people that competitive employment is possible. - '* 

2- If this approach does not work, issue a directive limiting 
the number of clients with non-competitive goals. Have 
counselors tell applicants they cannot receive services if 
they do not try for competitive employment • 

Conclusions • ' 

These two examples show how the standards information, combined with 
investigative logic, can enable managers, assisted by their program eval- 
uation units, to hone in on the specific problems reflected in general ^ 
program indicators. Supportive evaluation is a .term that covers the 
activity of using standards information and other program information,* 
to da analysis using program data to support management decisions , in 
order to answer questions about the state's performance in the provision 
of rehabilitation services and to improve that performance. Through the 
creative use of this approach to employing program information, managers' 
and evaluation staff can work jointly in t.he identification of practices 
and environmental conditions affecting performance. 
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VII. PRETESTING THE PROCEDU RAL STANDA RDS: 
BACKGROUND ON THE STANDARDS, PRETEST 
PROCESS, AND ANALYSIS OF PRETEST DATA 

^> 

This and the next two chapters report on the pretest activities involv- 
ing the four Procedural' Standards • The Procedural Standards are treated 
separately from the Performance Standards since they pertain to service 
methods and processes, rather than service outputs and outcomes; and because 
of their different data requirements • But our basic concern in these chap- 
ters is the same as in Chapters IV and V: to develop recommendations for 
a refined system of Procedural Standards data elements, based on synthesis 
and consideration of the comments received from all our reviewers. The 
proposed Procedural Standards are as follows: 

Procedural Standards 

9. Information collected on clients by the R-300 and all data reporting 
systems used by RSA shall be valid, reliable, accurate, and complete. 

10. Eligibility decisions shall be based on accurate and sufficient 
diagnostic information, and VR shall continually review and evaluate - 

, eligibility decisions to ensure that decisions are being made in 
accordance with laws and regulations. 

11. VR shall ensure that eligibility decisions and client movement 
through the VR process occur in a timely ^manner appropriate to the 
needs and capabilities of the clients. ^ 

/ 

12. VR shall provide an Individual ized Written Rehabilitation Program for 
each applicable client, and VR and the client shall be accountable to 
each other for complying with this agreement. 

IVe shall present a thorough discussion of each of these standards, 
including: 

• The rationale for each 'standard; 

• The data collection methodology and the process for analyzing the 
data ; 
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o A synthesis of comments received from the MEU's, program 
evaluators, and reviewers, along with our responses and 
recommendations in light of the comments; 

9 Analysis of the pretest data provided by the San Diego State 
University Case Review Project; and 

d Summary recommen4ations for the system design. 

Before we begin discussion of the individual standards, softie back-x 
ground is needed into the overall Procedural Standards system. In this 
section we discuss a variety of issues relevant to the Procedural yotand- 
ards pretest. First, we discuss the standards themselves: wha'L they are; 
why they are needed to facilitate achievement of RSA's broad g</als;^ and 
the data sources used to assess performance on ^the standards, ^ext, we 
discuss the general process by ,which the Procedural Standards data was 
collected, and^ the time investment required to collect the data. Last, 
ue discuss general issues regarding the analysis and interpretation of 
the pretest data. 

THE PROCEDURAL STANDARDS 

The Procedural Standards we have recommended within the standards 
system are intended as 'a method of ensuring attention to three critical 
process areas, and to da-^a validity. It is intended for' states to use 
• the Procedural Standards to benefit their program evaluation efforts and 
facilitate the improvement of services to clients. These procedures will 
form the basis for agency decisions to make appropriate changes in prac- 
tices, where current processes are not in keeping with client interests 
and positive program performance. 

The recommendations for the Procedural Standards reflects the desire 
..to allow maximum flexibility to states in the .VR process, yet still ensure 
attention to the areas addressed by the Procedural Standards and provide 
sufficient data in the^e areas to allow for programwide analysis. Ideally, 
a uniform procedure would be followed by all. states for monitoring these 



As" discussed in CJiapter III. 
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process areas, even though states retain differences in the ways they, organ- 
ize and conduct case service delivery. Indicators of compliance with legal 
requirements, such as' eligibility and IWRP, should be the same for all 
states, i.e., the same questions should be asked and the same summary data 
% should be reported. 

Most of the needs of the Procedural Standards are best met through 
case review ♦-^ Thus, we are recommending that a single case review process 
be implemented to address the case review needs of all four of the Procedural 
Standards.. We recommend that the Case Review Schedule (CRS), developed by 
the San Diego State RCEP IX, be used as the basic document for Procedural 
Standards d^a collection. The CRS has already been mandated by RSA as 
the standardized instrument to be used by regional RSA offices whenever they 
conduct case reviews."'' F6r Procedural Standards 10 (eligibility) and 12 
■ (IV/RP), BPA has selected the CRS items which we consider essential tc\ 
adequately assess compliance. These items make up the Modified CRS, which 
is cfonsiderably shorter than the full CRS, RSA could choose either the CRS 
or the MCRS' as the instrument for collecting Procedural Standards data. 

^iVhiJe the CRS is an^appropriate vehicle for collect iitg compliance, 
data, it lacks certain items needed to assess the validity of R-300 data 
(Standard 9) or to assess timeliness of case service "(Standard 11). For 
these standards, BPA developed separate instruments to complement the CRS.^ 
^ If the Procedural Standards are implemented, these two instruments would 
be incorporated directly into fhe CRS. to provide a unified data collection 
instrument. ^ ' . ' I 

Having described the general thrust of the .Procedural Standards and 
• the general process fo^ collecting the needed d^-ta, we turn ne?ct to a 
discussion of the individual standards. 

, \ 

STANDARD 9: R^300 DATA VALIDITY A^fe 'RELIABIHTY ; , ■ 

Information .collected '1(^,11 clients by^ the R-300 and all data 
reporting systems 'US^d ;by RSA shall be valid, reliable, 
accurate^ and complete. ' 

Tlic VR service delivery systems needs an objective data base from 
which to measure performance., Yet inconsistencies and errors in reporting 
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Currently exist among and within VR program data systems. Confusion or 
misunderstanding over definitions exist and need to be minimized. This 
Procedural Standard would ensure ^that state agencies maintain acceptable 
levels of validity and reliability in reporting of R^300 and other data* 
This standard assumes states' attention to good data processing is pertinr 
ent to all the standards* Thus, given the importance of reliable, valid, 
and accurate data on which to base the program's evaluation capacity, we 
feel that this Procedural Standard relates to all of the broad RSA goals: 
compliance, quality, and cost-6f fectiveness . 

Reliability, accuracy and completeness* of data should be checked in 
several ways. t*/hile we would recommend validity studies on a periodic basis 
and edit checks as a part, of routine data processing, this standard en- 
compasses a specific rec^nunended procedure for states to follow to ensure 
^ the accuracy of data recorddd and submitted to RSA through the R-300. 
Primarily, the case review process should include an accuracy check between 
the case folder information, the R-300 form itself and, if the state has a 
computer system, computer output listing of R-300 items selected for review. 
In particular, those R-300 data items which ajre ;used in computing the stan- 
dard's data elements should be subjected to "Jmecks of accuracy and validity 
through case folder documentation. 

Further checks on the accuracy of the data recorded can be made by 
verifying the information recorded in the case folder with the client, or 
through other souces (such as employers, public assistam:e agencies, etc.), 
in order to determine whether the dataycecorded are, in fact, true. While 
we feel this is important to the accuracy of the data, 'it would be expen- 
sive and time-consuming to conduct such a verification procedure through 
routine follow-up or personal contact. Our recommendation in this area is 
to include such a check in a demonstration project to see whether a '^reality 
cheeky would in fact significantly increase the accuracy of the R-300 data. 
If it does not, states should not be expected to continue the verification 
process on a routine basis, but expect that such information verification 
will be done periodically as part of RSA's, central data validation prDce- 
dure . 
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STANDARD 10: ELIGIBILITY' DETERMINATION 

Eligibility decisions shall be based on accurate and suffi- - 
cient diagnostic information, and VR shall continually review 
and evaluate .eligibility decisions to ensure that decisions 
^re being made in accordance with law s an^ regulations, 
i ~ 

The determination of an applicant's qualifications for eligibility is 
a critical point in the Vk process for both the client and t^. agency, 
this standard seeks, to protect client interest by requiring state agencies 
to install procedures for monitoring eligibility decisions in a sample of 
cases and ensuring that the decisions are appropriate, in compliance with 
legal requirements, and supported by the proper diagnostic information* 
This standard pertains to two of RSA's broad goals. First, inasmuch as 
the eligibility determination process rests on a legal footing, the standard 
pertains to the goal of compliance with the legislation. Second, we feel 
that it pertains to the goal of cost-effectiveness, since it is a misuse of 
money to s^rve ineligible persons, particularly if other, eligible clients 
are turned away due to an incorrect * determination o£ ineligibility. 

In establishing a procedural standard for the review of eligibility 
determination we are concerned with the appropriateness of the decision and 
its accordance with laws and regulations". We expect :^nformation from this 
review to address two facets af this concern: (1) that clients who are not 
eligible for VR services not be accepted for services, and (2) that clients 
who are eligible are indeed accepted. 

IVhile monitoring and review of eligibility decisions by supervising 
counselors or managers will provide a check' on that determination, states 
have varying supervisory structures and roles and should be allowed to 
retain flexibility in their monitoring practices. ' Although we support a 
cross-check on eligibility decisions, we are not recommending its inclusion 
as a requirement for this standard. 

Our primafy recommendation is for the use of the case review system 
to monitor that decisions are made in compliance with the standard. We 
have recommended using the CRS as the review instrument , including the 
appropriate items addressing legal requirements for eligibility determina- 
tion. Those relevant items from the CRS do in fact quite adequately cover 
the process concerns reflected in this standard. 



STANDARD 11: TINffiLINESS 

VR^shall ensure that eligibility decisions and client move- ' 
meQt through the VR process occur in a timely manner appro-^ 
priate to the needs and capabilities of the clients . 

This standard seeks to avoid delays in the VR process that are likely 
to impede or hinder successful rehabilitation of, the client. Rather than 
set a performance standard using time-in-statuS to define ••undue delay," 
this Procedural Standard requires that each stateTiave a monitoring or 
flagging mechanism for cases remaining in statuses over a given- length of 
time, and a procedure to evaluate the appropriateness of any case delay. 
Many of the state VR agencies already have variations of such a system in 
place. 

We include this standard among those pertaining to the RSA. goal of 
providing quality case services, for two reasons. First, one aspect of the 
quality of a client's service experience is the speed with which his or her 
ease is handled: did the client feel that the counselor ''cared" about 
him/her (as evidenced by the fact that the counselor "kept on top o^ 
things" and "kept things moving along"), or did -the coun^lor seem to put 
him/her on a' lower priority? The client's perception of his/her treatment 
by VR can have an impact on his/her attitude toward VR and about the 
usefulness of participation in VR. Second, research on successful rehabili 
tation outcomes has suggested a relationship between timeliness and success 
(perhaps as a consequence of the perceptions .discussed above) • 

ThC' issue of timely^ case movement or "undue delays," as it is phrased 
in the current standards, has been one of long discussion and controversy. 
IVhile there is literature t<> support the correspondence between certain 
times in process (particularly time to eligibility , decision) and outcome, 
there have also been questions about inter-rater reliability in the area 
of judging timeliness of case movement through case review. Nevertheless, 
an overall review of timely case movement on a client-by-client basi;5; is 
best handled through case review, if items can be identified which/hatve 
good inter-rater reliability. . / 

• " / ' ■ 



• Much effort has gone''' into attempts to <iefine.; arid establish ^s.tandards 
for, timeliness .of case service progress* As not|d, research -on success- 
ful rehabilitation butco'mes has supported the concern for timeliness 'in 
Establishing a relatioT^ship between the time required for 'an eligbility, 
decision and ultimate client 6utcora^. Previous attempts to monitor the 
.timeliness of service ptovision by way p£ a /standard, on "undue delay^'*have 
b6en hampered, however, by 3everal problems. The first 'is the definitional 
and reliabinty problem, "Undue deVay" and its converse, "expeditious", or 
"quick afid effi^ient-'^ case management, mean different things to different 
people.^ , The current standards ' use the approach of arbitrary time periods 
to define "timely" case movement: eight months has been defined as the 
limit fpr timely eligibility decisions; 22 months for Jimely completion of 
the VR process. This approach ^las been widely and justifiably criticized < * 
for its lack of sensitivity to the legitimate differences , in individual 
cases: a complex case, perhaps involving long-term educational services, 
might well require more than 22 months, without any delay, IVhere a case 
has been subject to a delay, jthe situation is further complicated by the 
differing implications of different causes for delay: lacJc of client res- 
ponsiveness^ inattention* or inefficiency on the part of the counselor or 
th* VR agincy, and problems outside of VR (failure of a vendor to deliver, 
ujifavoidable waiting lists in training programs). Each imply very differ- 
ent responsibility for time lapses and cannot equally be ascribed to VR 
agency "failure," 



thus, use of "objective" measures 'of timeliness has suffered from arbi- 
trariness and frequent inappropriateness of established time cut-offs for 
many clients. Other approaches to objective measurement, such as recording 
planned initiation and completion dates for each service, and monitoring 
compliance with the* scheduL^, suffer from cumbersomeness in execution. On 
the other hand, subjective judgments df timeliness have been vulnerable to 
criticisms of unreliability in applicatipn. BPA believes, however, that 
this unreliability may well have arisen due to the incorporation of two 
distinct concepts into the previously used "undu5" delay" judgments. ^ This 
term, "undue delay," includes concepts of bothjtime lapse and judgment of 
blamcj culpability or unjustifiable time lapse. (The word "delay" itself 
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sometimes connotes willfulness or negligence, and the modifier '*undue" 
definitely implies such problems*) Case reviewers might well differ in ' 
judgments as to, the cause of a delay, .and thus whether VR should be held . 
accountable; and for this reasons, reviewers may differ in their classifi- 
cation of a case, one citing an undue delay, another seeing an unfortiinate 
time lapse, but being unwilling to label it an undue delay if client moti- 
vati^on o,r outside vendors played a role. * , 

In response to the problem of a dual focus in assessing timeliness, 
we have proposed and performed preliminary testing of an approach which, 
, relies upon reviewer* judgment , but which divides case assessments of time- 
liness into two segments: first, a notation of whether delay has occurred 
in terms of time lapse between necessary activities in a case; and, second, 
an assessment of the reasons for the lapses."^ The relevant questions are 
appen?(j^ to the Case Review Schedule, and concem.jcritical phases of case 
progress eligibility determination; development of servi^ plan; and . 
service delivery and termination. 

This review should take place for a sample of closed cases in each 
state. IVe have also proposed that a case flagging mechanism be required 
to address cases still in the open caseload to monitor their movement 
through the process as it occurs. This -Procedural Standard 
would be in the form of a mechanism to be set up .by each state to flag 
each case which has remained in a given statifs longer than a specified ^ 
period of time. ■»'•,' ^ 

Obviously, an effective, model flagging system needs to be developed 
for: use by all states in response to /Standard li. Such a system might 
provide a signal to counselors in the form of a mori'thly client list mth 
times-in-status when clients had been in a given status more than a stand- 
ard period of time. Review of the client's ^it.u^tion should then take 
place (in a format decided by the. state) to* deteriSfhe' if case movement is 
appropriate, but no reporting* to RSA would be., required. Ideally, the amount 
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of time-in-status beyond which flagging occurs would be set at appropriate, 
even if different, times for each status. However, even before the flag- 
ging system is developed, research is needed into the time levels which 
are appropriate for consideration as general limits for time in pa]?ticular 
statuses. As part of our pretest of the standards, BPA undertook a first 
cut at this analysis, using data from the timeliness assessments and from 
the R-300. A summary of our system development appears later, in the dis- 
cussion of the Timeliness Assessmgnt instrument 

' STANDARD 12: IWRP ' • ^ . 

VR shall provide an Individualized Written Rehabilitation 
Program for each applicable client, and VR and the client 
shall be accountable to each other for complying, with this 
agreement • 

There are several aspects of the Individualized Written Rehabilitation 
Program that would be addressed in this Procedural Standard: (a) compliantj 
with the requirement that an IWRP be fully developed for clients accepted " 
for services or extended evaluation; (b) assurance of the protection of 
client rights and client awareness of the remedies available for mitigating 
dissatisfaction; (c) joint client/counselor development of the job goal and 

^ the service plan; (d) mutual client/counselor responsibility for follow- 
through on the agreement and annual review of its progress and appropriate- 
ness; and (e) the appropriate handling of plan revisions. 

This standard bears a relation to the RSA goals of compliance and qual- 
ity case services. Obviously, given the regulations mandating provision of 
an IWRP to all accepted clients, this standard's relation to the compliance 
goal clear. While the regulations concerning the IWRP stipulate compli- 
ance with the provisions of the law, elevating the issue to the level of a 
procedural standard will ensure compliance with the legislative intent of 

, the IWRP. 



Tlie full analysis leading to our proposed model can be found in 
BPA's report. Review of State VR Agency Procedures for Case Flagging 
and Quality Assurance (Septepiber, 1981), available from RSA. 
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Inclusion of this standard could be justified simply on the basis of 

the strong regulation regarding compliance with the IltfRP provisions of the 

1973 Rehabilitation Act; However, perhaps an even more important reason 

to include this standard is the fact that research has shown a positive 

association between compliance with the IWRP requirements and successful 

1 ' - 

outcomes of the VR process. Since research has supported the premises 
underpinning the IWRP by showing that the process and the possession of the 
IWRP affect client outcomes positively, adherence to the IWRP requirements 
becomes a powerful norm for quality case management in VR, as well as a 
protection of client interests and rights. 

With the IWRP, as with the standard on eligibility, we are Tecommend- 
ing using the CRS to check for compliance with legal requirements for the 
plan. The Case Review Schedule includes a thorough set of items address- 
ing this issue, and we have used it in our pretest. Unfortunately, while 
anticipation of a case review of closed cases should provide incentive to 
service delivery personnel for complying with these requirements, inadequate 
planning discovered after the case is closed will not benefi*^ the client. 
If the case review system results in a given caseload being reviewed only 
once every two or three years, many open cases may be closed before the 
review can take place. Therefore, states may find it useful to institute . 
an in-process compliance check on at least a sample of open cases, while 
there is still time In the process to improve any deficient plan. This 
goal can be accomplished by the inclusion. of open cases in the uniform 
case review process, 

DATA COLLECTION PROCESS FOR THE PROCEDURAL STANDARDS 

Three major pretest activities were required to respond to the four 
proposed Procedural Standards, The three pre'test activities, which in 
operation were combined in a variety of ways, included: 



^Berkeley Planning Associates, Implementing the VR Act of 1973:. The 
VR Program Response , p. 59. (1978) 
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. Case Review Schedule designed (Originally by San Diego State 
University RCEP IX tq d^ss^ss states' compliance with the 
Federal Regulations and modified by BPA (in. form and in analysis, 
althtmgh not in item content) to collect information on. client's ' 
eligibility determination and IIVRP proceisses, in order to respond 
to Standards 10 and 12. 

/ 

• R-30Cr Verification procedure designed to assess the degree to 
which information submitted Xo \Sk on critical items of the 
R-300 (defined as itefins used in the calculation of Performanc-e 
Standards' data elements) was corroborated by casefile informa- 
tion, in order to respond to Standard 9. ' > 

• Timeliness Assessments designed to link subjective assessments 
of the. timeliness of case'mpvement to objective data on the* 
length of time spent in various statuses by different disability - 
types, thereby' generating a data pool from which accountable 
parameters for times-in-status could be drawn, in order to 
respond to Standard 11. * ^ ^ 

Again, it should be noted that when fully implemented, these three 
instruments would be integi^'ated into one instrument for use in collecting 
data on the Procedural Standards. 

1 ' ' 

Data Collection ^ \ 

Two observations should be made regarding the overall process for 
data collection;. 

• Three different approaches to collecting the limited CRS data 
required to respond to the standards were employed: {1) selec-- 
tive analysis of the requisite subs'et of data items collected 

on the San Diego -Instrument (conducted in all six MEU's); ^ * > 
(2) utilization of a modified CRS instrument on a small sample 
of cases in two MEU's (Oregon and Pennsylvania); and (3) pre-" 
testing of an integrated instrument on six reliability cases in 
^ Mississippi in which data items for all the Procedural Standards, 
as well as case quality indicators, were included. 
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r Also, Mississippi's experience as the initial pretest of the 
R-300 Verification procedure revealed serious deficiencies in 
the answer scale definitions and training instructions. The 
resulting responses wer*e highly unreliable, as manua^lj// analyzed 
on-site. As a consequence, the activity was terminatBd, the 
, instrument and training revised, and a return training visit to 

Mississippi, after the other MEU's had completed their data col- ' 
4;^ection, ^resulted in completion of the data collection effort. 

The sample dimensions achieved for each pretest activity are presented 
in Table 3,8, which also illustrates the previously mentioned variety of 
combinations of the three activities in implementation. It is important 
to remember that* the samples of cases selected for the San Diego CRS in 
each MEU were used throughout the* Procedural Standards for the other 
pre-test activities. Ideally this sample would have been extended, into 
those activities in the Performance Standards requiring 'sampling (i.e., 
R-500 Supplement C§D, Closure and Follow-up Surveys), in order to 
optimize supportive evaluation opportupities on the same cases. However^, 
the duration of the pretest and the*^"^ independent logistical planniitg of 
SDSU and BPA early in the pretest permitted common samples on only 60 
26-cl05ures and 60 28-closures from Pennsylvania. Mo^t unfortunately, " 
the samples contain no 08 closures. This precludes any analysis of 
compliance with regulations designed to protect the rights of clients 
closed as ineligibile. • - 

¥ 

Time Requirements 

The labor investments tq complete the Procedural Standards are pre- 
sented in .Table 39. The average lievel of effort, including Program 
Evaluation and support sta^f, was jl57 person days, or nearly two-thirds 
of a person year, to administer , prepare, train, execute and'analyze 
the Procedural Standards pretest activities. By far the greatest propor- 
tion of time was consumed by the San. Diego fRS which, on average, accounts 
for 80% of the effort. The wide range for these activities from 117 
person days to 216 person days -- can be explained in part by which activi- 
ties were comKlnId in training and execution, as well as by the number of 
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Table 38 

Procedural Standards Sample Dimensions 



♦ 


DEL 


MICH 


MISS 


OREG . 


PENN 




Full SDStI CRS 
Status 28 


120 
. 60 
60 


120 
60 
60* 


120 
90 
30 


*100 
50 
50 


110 
55 

55 - 


120 
•60 

60 


Status 26 
Status 28. 
Status 50 
Status 08 






-- 


20 
5 
5 
5 
5 


20 
5 
5 
5 
5 




CRS Plus Timeliness 








120 


150 




R-500 Verification Alone 




120 




120 


120 




R-500 Verification Plus 
-Timelirjess Assessment 


' 50 




12p 






120 


Timeliness Assessment 
Alone 




120 : 








-J 


Procedural Standards 
Case Review Instrument 






6 
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Table 39 
Procedural Standards 
Pretest Level of Effort 
. (in person days) 



p. 


DEL 


MICH 


MISS 


OREG 


PENN. 


VIRG 


AVG 


General Adjninistration 


28.0 


5.0 


.5 


1.2 


27.8 


2.3- 


10.8 


R - no V<=»r 1 -Ft r a 1- i nn P 


0 . U 


a 

2b . 0 


b 

1 o r\ 

18. 0 


10.0 


25. 0^ 


18.0 


17.3 


Timeliness Assessment 














Training 


2.0 


6.0 . 


- 4.0 


2.0 


10,0 


4.0 


4.6 


Execution 


6.0 


19.0 ■ 


14.0 


8.0 


15.0 


14.0 


12.7 


San Diego Case Eleview 


179. O'^ 


103.0 


99.0 


105.0^ 


139.0^ 


13a. 0 


125.8 


Schedule 
















Preparation 


40.0 


15'. 0 


10.0 


17.0 


20.0 


25. a 


21.2 


Training 


68.0 


44.0 


42.0 


36. P 


70.0 


44. O/ 


50.7 


Execution 


50.5 


33.0 


36.0 


36.0 


36.0 


48.0" 


39.8 


Analysis 


20.5 


11.0 


11.0 


16.0 


13.0 


13.0 ' 


14.0 


Miscellaneous 


l.'O 


0.0 


0.0 


1.0 


5.5 


2.5 


1.7 


(Case Flagging § 
















Case Quality) 












# 




TOTAL 


216.0 


13-3. 0. 


117.5 


117.2 


197»3 ■ 


152.8 


155.6 



Two separate groups reviewed the and timeliness aspects of the 

procedural standards in Michigan. Four PE unit staff completed the 
timeliness assessments; ten district office managers completed the 
R-300 verifications. 

) > » 

These are the values for the second data collection effort in Missis- 
sippi. The initial effort resulted in substantial revisions iri the 
data collection instruments and instructions.^ 

'In Oregon and Pennsylvania, the R-300 Verification procedure was con- 
ducted separately; the Timeliness Assessments were included in the SDSU 
Case Review procedure. 

Wlaware completed the CRS prior to the initiation of the pretest and 
subsequently elected to continue the process as an on-going evaluation 
activity. The l^bor estimates include both the initial CRS training 
and data collection effort, as well ^s two follow-up training activites. 
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reviewers in each MEU, In addition, Delaware conducted two advanced train- 
ings of all the state's counselors following dissemination of the results 
of the initial Case Review activities, thereby substantially increasing the 
level of effort estimates in that category.^ 

The Case Review Schedule is of particular interest when considering the 
issue of time requirements, inasmuch as this is the longest, most detailed, 
and most demanding of the three Procedural Standards data collection 
instruments. The data collection methodology stipulated that 120 cases, 
reflecting aspects of interest to the state (e.g., various closure statuses, 
district or regional differences, urbW versus rural areas), be selected 
for each MEU, The level of ettort required to complete the CRS on these 
120* cases is primarily a function of the number of reviewers involved 
times a maximum of five days for training and reviewing. Other variables, 
which can affect the total. labor investment, presented in Table ,39, 
are: in-state organization of casefiles, appropriateness of selected 
reviewers, effectiveness of trainers in consensus building, and number of 
attendees at the foriow-up training and presentation of results. Only 
Delaware's investment of 179*person days far surpasses the average of 126 
person days for this activity. This is the result of two additional 
advanced trainings subsequent to the initial CRS training and data collec- 
tion effort. Overall, the Case Review absorbed the greatest amount of 
effdrt. On average, it accounted for 44% of the total labor expended in 
pretesting the proposed Program Standards (i.e., both the Performance and 
Procedural Standards), • 

In contras.t, and as can be sensed by the data in the table, pretesting 
of the R-500 Verification ^jad. the- Timeliness Assessment instruments required 
far less tine. On averag-^^, 17 person days were required to complete these 
pretest activities on the 120^c?ise sample. However, these figures must be 
seen in the context of their role vis-a-vis the total Procedural Standards 
data collection process,, That is, although these instruments request 
information of a different kind than does the CRS} in the ideal situation 
they would be completed .in conjunction with the CRS, This^is because the 
three instruments compl^knt each other, both in terms of 'the information 
they elicit, and in terms of the overall understanding provided to reviewers 
•^hen the instruments. are used in conjunction. The point is this: it is 

X- ■ ■ :i 
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somevvhat irrelevant that the CRS requires so much more effort to complete, 
relative to the Timeliness and R-300 Verificatigp, instruments; the three 
should be seen as a package. 

Further, while the CRS did require an exceptionally greater amount 
of time relative to other instruments, we feel that time requirements for 
the CRS (as well as all the other instruments) will show a great reduction 
when the full system is implemented, because: 

• reviewers will become more efficient with experience, and 

• problems pointed out in earlier reviews will be corrected through 
training and feedback, thus improving case documentation and 
facilitating the process of case review, ^and ultimately reducing 
effort required. 

ANALYSIS AND INTERPRETATION OF^I^RETZST DATA 

;se present the results of the pretest data in the next section of; 
this report. Prior to this presentation, however, we wish to make the^ 
reader aware of the .special issues surrounding the pr.etest data, and the 
use of that data. To start, we must always keep in mind that this is a ' 
pretest effort; it is not .an actual assessment of agency performance. 
As such, while we do discuss -our findings with regard to agency performance, 
we focus on general performance trends which can be observed across states, 
rather than focusing on individual states* performances. (By ''across' 
states," we generally mean at least two states showed problematic 
performance). Further, we attempt to suggest the causes of and possible 
solutions to any problems observed; again, in terms of all states rather . 
^ than individual states. IVhen implemented, the standards data will be used 
to identify problems in and target tradnihg and assistance to individual ' ' 
'states . . • . * 

Tfie second point is that-, given the pretest nature of this activity^ 
we are concerned with the quality of the data itself: to what degree can 
we "trust" what the data tells 1^3 about agency performance? As part of 
the pretest we wish to know those items which provecj /problematic from 
M'n interpretation standpoint due to inability of reviewers to arrive 
at I consi.^tent understanding of "success": that is, to arrive at common 

• ' '.. • 26.3 . . 
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understanding of what does and does not constitute' adequate document a tj^,. 
compliance7~vaiidity, timeliness, and so on* Onpe we know the problematic 
• items, one of two things should ^happen: \^ 

• the item should be deleted from the data' collection ipstrumenlf; or 

• reviewers should receive enhanced training (informed by use and 



discussion of the instrument) to overcome the consistency prob 



lem«. ^ 



Our preference is that the latter process occur, We-^iegl^nhat ,^ as 
designed, the data items included in the Procedural Standards reptesent 
the minimumrneeded to assess compliance, data validity, and timeliness. 

The San' Diego RCEP has developed a technique for assessing inter- 
reviewer reliability. Their technique,^which results in a ^'consistency ^ 

4 

ratio" for each item, appears in Exhibit 7, We include the consistency 
ratios resulting from .the pretest data in the various 4ata tables in 
Section VIITof thiv-^«^ort. 

Finally, the RCEP has 'established certain criteria fOT interpreting 
the results of the data. These criteria pertain to , ' 

• the minimum number of cases needed before data can be analyzed*; 

• the interpretation itS^lf '(i»e., ^iven a certain , level Of \, 
performance, should tjie agency mave to correct the problem 

^ immediately, should they sipipiy ''consider action," or what); 

and ^ / , . - - 

• the necessary level of consistency among reviewers needed to allow 
data interpretation. \ . . - 

The si>ecific criteria for the first two issues are presented in 
Exhibits 8 and 9/ As regards. the third issue, a -75 consistency ratio 
has been established as the minimum level necessary for unquestionable 
data interpretation fpr each of the sections of the BPA/M^R. .IVhile a. 
minimum^lev^e^i'of .75 is statistically sound, it is an arbitr.ar7 figure 
and canvie raised or lowered the discretion of RSA. We feel that .75 
is appropriate. We tak^ issue, however, with the RCEP's ctiteri^ for ' 
r-ecommending corrective action. As Exhibit. 9 shows,.a level of -75% 
compliance is deemed acceptabl^V'^Ve would 'raise this to 85% - 
since the purpoi>e,of the Procedural Standards fs (among other things) , 
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to promote maximum ct)mpliance with regulations, data Validity, ^d timely 
case service. , • ^ * ^ ' . 

„ Kith these general points in mind, we next turn to a discussion of 
,the comments and data collected during the pretest. 
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Exhibit 7 

* ^ 

/ ^ ... 

1 

PROCESS OF DETZFJ^miiyG TES HTZFJiATEE HEZIABIUTZ ' (consiszency) RATIO 
EXAMPLE OF 

CXS, IIA, ITZM n (BOOXrZT CODE NUMBERS ARE FICTITIOUS): 



(There must he a rniTiimum of 3 
reviewers and a xazi^jjn of 12. 
The excjnple shewn, here indicates 
9 reviewers) 





"r 


♦SCOKLET' 


REVIFrfER . 


MUMBER * 


• 




* CODE ' 


.1, 2 3 4 


3 


6 7 3-9' ■ 




1 


■♦029-237* 


11 1 1 


1 


1 1 1.1' . 




2 


*061-Q69* 


L 1 1 1 


2 


1111*. 






*073-0So* 


1 l*-! 1" 


2 


2 ,1 1 2* . 




i 


♦101-10'9* 


1111 


1 


1111' 


• 

{ 


3 


*U6-144* 


112 1 


1 


1112'.^ 




5 


'159-167* 


1111 


2 


I'l 1 1*.. 



Responses: • l=Yes; 2=Mo; 3= Not Applicable 

. Reading acrcb the zov of Case 1. the #1 response is the highes' occurring 
'response or node. In this ca^e there are 3 responses or .^1. m the roi- 
loving cases the 3ode is: * 





2; 


#1 


is 


zhe 


mode 


with- 


8 


responses 






Case 


3 : 


^1 


is 


the 


mode 


with 


3 


responses 






Case 


4: 


^1 


is 


the 


mode 


with 


7 


responses 






C2ise 


5: • 


#1 


is 


the 


aode 


with. 


7^ 


responses 






Case 


6 : 


r 1 


is 


the 


mods 


with. 


3 


responses 




4 



A consistency ratio was delrived by combining the total nxaiber of responses 
for each aode Ln each case and then dividing by the total nuaoer or actual 
responses, _ ' ■ 

Nune--tor =3*3 6 ^ 9 * 7 ^ 8- = 46 QNuaber of re-rL swer s ' in agreesient) 
beno^atcr 9 x a-.piunDer o f response^ = nuaoer o; revxewers X nmaoer or cases) 

OR 

io _ = ,33 ■ . • ■ ■ 

*, 9 X o ' 34 ' ' " ■ ♦ ^ ■ 

. A .73 consistency ratio has been established as the aini::nm level necessary 
for .jKCuestionabie data Lntarpretation for e^ca section or tne C?S. 

Q ^From 5DSU0 RGEP report produced in the Volume I Pretest Report, page 295. 

1^ . . i 267 ' . ' / 
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Exhibit 8 



Criteria for Referring D^ira Resuirs on Each Question 



Percent of Cases 
with active res- 
ponses on each 
item, y 


Status 26 
(N-4a) 2/ 


Status 2Si*- 


Outcome ♦ 


Discussion 


Si's and Above 


COS es 


N=49-2S 
cases 


Reported j^ithout 
special consider- 
ation • 


Both open § closed statu 
has 51% or more active' r 
ponses on a question. 


21 - S0?5 


N^2S-11 
cases 


N=2S-11 
cases 


Data reported 
vdth special 
consideration. 


Either the open or close 
statxis was between 21% S 
50% of the active respon 



I 
I 
I 
I 



20% • Below 



:tivV re 



M=Less 
than 11 
cases 



N^Less 
than 11" 
cases 



Data not report- 
ed in this 
analysis.. 



Either the open or closed 
status was below the 20%| 
miniaun. The results ve| 
not reDortsd. 



1/ActivK responses refers to the total percentage of responses, , excluding M/A 
responses on each It en. 

2/The nunber of cases reflects the total cases in either Status 26 or 23. 
These are referred to as, unduDlicatec cases. 



1 
t 



1, 



From Preliminary Pretest Report, page 296. 
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Exhibit 9 



Scanaards for Inrerpretation of Percentage of Responses 





[ 

— '«»ntage of Active 




Response rorniat 




[ Re 


sronses oer Itea ^ 


''Yes'' o,r "Adeqtiate" 


"No" or "Less 
Than Adequate" 


Discussion ' 


* 0 


- 2S'i 


— w - 

No Comment 


No*. Coinment 


There is no identifiabl 
basis for conraent* 


1 25 


- 30^ 


No Connaent 


Area of Concern: 
E:ca?nine 


Provide closer examina- 
tion of results and 
* consider action. 


3i 


- 75'} 


No Conment 


Attention: Con- 
sider Action 


Explore recomaendation. 
and develop a plan of 
remediation. 


j' 76 


- 100% 


Action: Posi- 
tive CoRimendation 


Action: Provide 
immediate atten- 
• tion 


Tais ^area requires iiEr. 
diate action in the ?N 
"Less than Adequate'5 
responses . 




^From Preliminary Pretest Report, page 296, 
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VIII- ANALYSIS OF THE' PROCEDURAlV^ANDARDS DATA . , 

This section presents the data obtained through the iinplementati'bn 
of the Procedural Standards pretest in the six MEUts, ike analysis .of the 
data will appear in thr^e groups, according to the dat^ collection metho- ,^ 
dology employed for each standai^d. The three groups are: 

Cas^ Review Schedule, which is the method of collecting data 
for Procedural Standards 10 and 12; . - 

• R'300 Verification, for Standard 9; ^ ' . '^ 

• Timeliness Assessment, for Standard 11. ' ^ 

For each of these three groups, the format of the discussion is ^s 
follows: ' . r ' 

• The rationale for the standard will be presented. ' . x 

• Presentation of the training and logistics lieeded to implement the 
pretest will then follow. . . 

• A. synthesis of the coifunents received from the MEU's, program 
evalu^tors, and reviewers will then be presented. .These comments 
will be grouped according to majoi* topics,, stich as **cost and 
effort'* and ''possible' benefits . " These comments were obtained 
mo'S.tly during training, through written evaluation formsi^ 

% Following the .comments, we will respond to critical issues raised 
in that section. . ^ . , ^ 

.• Lastly, presentation and analysis of the data. , ^ 

Based on the 'information found in this section^ Chapter 'X of this report 
jvill present our recommendations for revisions in the Procedural Standards. 

■ ' .' ' \ ' 



■''Verbatim comments are presented in the Preliminary Pretest Report 
(Noyember, 1980).,-. ' ' _ t 
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CASE REVIEW SCHEDULE 

The Case Review Schedule (CRS) designed by San Diego State University 
RCEP IX responds to Standards 10 and 12, by assessing the states' compli- 
ance with federal regulations and by collecting information on clients' 
eligibility determinations and IWRP processes. In addition, BPA developed a. 
Modified CRS which deleted many items not necessary for assessT.ng states' 
performance on the Procedural Standards. 

The CRS, in both its official version and even in its modified version, 
is too extensive to reproduce in this section. It has been included in 
Appendix 2. In responding to Procedural Standards^ 10 and 12, lonly those data 
items (i.e., regulations) pertaining to the eligibility determination and 
IWRP processes were analyzed. 

Training and Logigtics 

Each state participated full); in San Diego's Case Review Schedule pro- 
cedure* ■ The steps in this procedQ^e were as follows: 

• Pre-trai,ning Logis^tics 

. -- Negotiate dates with each state 

Negotiate sample dimensions vitji each state 
Sample selection (l20 cases) 

Case files sent to central location for review 
Completion of Section I (identifying information) 
Review team "selection ('8-16 reviewers): 

* 2-3 counselors 

* 2-3 supervisors 

* 1-3 program evaluation specialists 

* 1-3 rehabilitation administrators 

* 2-3 regional office observers 

# On-site Training (2 days) 

» 

-- Five to six hours of training focused on orientation to CRS 
sections and corresponding instructions in CRS manual 
Individual application of CRS to representative training case ^ 
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— Small group discissions to process., results of, training case- 
on an item-by-item basis. 

— Discrepancy resolution in large group discussion 

. Repeat of individualized case review/ small group process, ' 
and large group process on second training case to further^ 
develop consistency' 

• Case Review (3 days) 

Standardized method of distribution of cases . 
Integrated distribution of six reliability cases to all 
reviewers to enable a determination of inter-rater consis- 
tency ^ ^ 

-- Case flow monitored throughout actual case review 

Several states (Delaware, Michigan, Mississippi and Virginia) were 
trained by SDSU prior to BPA's authorization to coordinate with SDSU on 
this activity. The two remaining states' CRS trainings were schedured 
for the same week in February, permitting BPA attendance at only one 
(Oregon). 

■ Issues Raised During trfiePretest 

The six MEU's were ask^d to comment on the Case. Review Schedule; 
on the genera l/^asibi^^ incorporating the CRS process into the Program 
Standards; and on specific problems they encountered and/or anticipate for 
'their* state, should the CRS be implemented to collect Procedural Standards 
data. We have grouped these comments into broad issue areas. Below, ive 
discuss the major issues addressed in the comments and present our response 
and recommendations. 

Cost, Effort, and General Benefits Derived From the CRS 

Reviewers raised the issues of cost and effort in relation to all 
three Procedural Standards data collection instruments. However, the Case 
Review Schedule generated the greatest number of negative comments among 
the. thre^ instruments. The specific issues raised include: 
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the instrument's length; 

• the staff time-resources required to conduct the case review, 
particularly if it were conducted annually (one state remarked, 
that the CRS "requires maintaining another audit team") ; and * 

• the potential "over -kill" of the instrument: as one ^view^ said, 
"It (the CRS) may be more detailed,, time-consuming, and critical 
than what is needed to evaluate compliance •" (Emphasis added.) 

Despite these comments (especially the last),. the CRS received quite 
favorable comments regarding the potential benefits resulting from, its use. 
The instrument was described by various reviewers as:* 

• "very comprehensive with respect to determining compliance," 

• "a thorough and uniform method of case review," and 

• "a long step towards^ the. development of a standaTized instrument 

to be utilized nationally which is^badly needed. " (Emphasis abided.) - 

Overall, tTien, the general feeling towards the CRS was that.: '(1) it 
is a good (and nee^ded) instrument for determining compliance with Federal 
laws, but (2) it requires too much effort'. 

Unfortunately, no provision^was made during the pretest data collection 
phase to test BPA's Modified CRS.^ This might have^pointed to a possible 
solution to the cost-versus-benef it quandry, since the. MCRS ijfses a fairly 
small subset of the full CRS questions. The data colleiCtofs should have 
u^ed the MCRS on a greater scale (only 40 out of a 'jxital 730 reviews used 
the MCRS5 , and they should have compared the effoj?4 required for the MCRS 
to that required for the full CRS. (IVhile this analysis may have been 
done, the results were not forwarded to us, nor did we even find evidence 
that separate records were maintained.) 

We might, in fact, have found no difference between the 'twS^'ins^ru- 
monts. In choosing the CRS item's needed for the Procedural Standards (and 
embodied in the MCRS), BPA strove to include those items deemed necessary 
to capture the/'essence" of eligibility and IWRP related issues. It may^ 
be that in order to "capture the essence" (a la the MCRS), one needsjrtJ^ ; 
know 'the details (a la the full C^S) . If so, the only functional 

.? ■ ■■ ■ . ■ ■■' . • 
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difference between the MCRS and the CRS would be the additional effort 
required to< actually record the details on the form; this may add only 
minimally, to the total effort • On the other hand, we might have found * 
significant differences in required effort. Unfortunately, we simply have 
no basis to makiS a judgement. «s 

^ Specific Benefits Derived ^From the CRS 

As discussed above, the CRS was generally viewed as capturing the 
relevant compliance concerns. In addition to this general benefit, several 
other points ^were made regai^ding a variety other, specific benefits 
potentially resulting from use of the CRS.- B^low we paraphrase these 
coniments. ' ^ 

• The' CRS^ helps provide a better understanding of eligibility and 
IV;RP regulations, both in terms of the content and the intent of 



regulations. It could be a good teaching mechanism for state 
agencies. 

• Prpvided that analysis re5ults feedback to field personnel, the 
CRS might result in better organized case folders containing more 
conplete documentation. 

« For management personnel, use of the CRS could alert the agency 
to particular casewbrlc problems,^ and identify areas where staff 
training is neede^. ^ ' . ^ ' ' . 

• ' The results coul/i help to revise and make new policy. 

• - The CRS pi/ovides a uniform, crbss-state evaluation of compliance. 

(Although, as (discussed below, certain advance procedures must 

« 

occur to ensure uniformity.) * 

The CRS'lformat lends itself to -'^segmental auditing'^ that; is, . 
to sepaprte rdviews of casework procedures during application, 
ex^i^d ^^^uation, etc. - * - ~ 

-parisoV With^ Exi.^ting State Case Review Methods 
Four of theJsix.MEU's reported that, in general, the CRS produced the 
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same results that is, pointed out the same strengths and weakness.es 
as were produced by the MEU*s own case review mechanisms. Not surprisingly, 
these stp^tes generally perceived no particular advantage to using the CRS 
other than [as stated by one state respondent) "as a training tool for 
federal regulations-*' 

' *^ - - * » 

Substantive Thrust of the CRS 

Reviewers in almost every MEU commented on what they perceived to be 
the misdirected focus of the CRS. Specifically, these reviewers felt that 
the CRS focuses too much on ^'paperwork" (i.e., the quality,,of counselor 
documentation) to thp detrimental exclusion of a .concern for quality case v 
handling. COne reviewer even remarked that the CRS ignored *'many areas 
of casework that should be covered/') The implica^jjj^ of these comments 
was that if the CRS is implemented, its orientation towards paperwork 
would **rub off** on counselors, leading them to stress documentation over . 
quality *case handling. In our opinion, these concerns are unfounded. It 
is true that, in general, the CRS items focus on case record documentation. 
However, in a sense, this is the way it must -be. First, we are dealing here 
with a case review process; that is, with a (generally) after-the-fact 
assessment of case handling. If such an assessment is to be possible, | reviewers 
must be able to find documentation in the* case record which establisliaB that 
counseloi^s followed proper procedure. » , " > ^ 

Second, the documentation sought by the CRS pertains to federal mandates 
concerning proper client selection-" (i.e. eligibility) , safeguards against 
improper determinations of ineligibility, assurance of clients! rights to 
* due process, and assurance of client opportunities to partij^ipate in r 
planning their service process, and objectives. We would argue that the CRS 
items are important not only b,ecaiXse they address issues of compliance with 
\ federal law, but also because they ^nanifest albeit indirectly a concern 
for quality casework and effective "^agency operation. For •example, the 
ineldgibility issue relates to coverage (i.^., maximizing service to the 
relevant population) and cost-effectiveness (i.e., not wasting resources on 
ineligiblcs).. As pertains to individual clients,* the IWRP-«:elated items 
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are even more directly concerned with quality case handling; it is 
generally accepted that greater client success results when the client 
feels he/she has played a role in shaping his or her' service goals. In 
short, we feel the CRS items do relate to quality casework (though they 
do not capture all aspects of case quality), while at the same time cap- 
turing important cdmpliance concerns. 

Finally, we ^are not worried about counselors shifting their focus from 
casework to documentation. Counselors will always fitst and foremost be 
concerned with the service process.. However, they must understand that. the 
environment in which they function particularly since passage of the 1973 
Act --^requires them to provide certain amounts and types of documentation. 
If they cannot (i^e., do not know how to) provide documentation, they should 
receive the needed training. (Basically, this would consist of teaching 
them how to describe .their actions and decisions clearly, and of impressing 
onthemthe importance o£ including backup documents.) If they will not 
provide the documentation, corrective action shoujd result. Likewise, if 
documentation doesn't exist because of failure to follow proper procedure, 
corrective, action is called for. tn^ short, we feel that documentation is 
merely the last step in quality case service; it follows as a natural re- 
sult. Documentatj.on does not become an end to itself. 

.Instrunent and Question- Forir.at .V,v 

■ — ~ ! — £ . .X 

Several comments were made regarding the instrument format and question 
wordings. Wo already mentioned reviewers' perce^ti^ons that the CRS is' too 
long. In additi^, various reviewer$ remarked that /the instrument format was 
''cumbersome and unwieldy, requiring huge .amounts of space," and that "the^ 
coding cards were a great nuisance." (The current form of the CRS uses 
"mark-sense" tards; the kind where ^2 pencils are us'ed and the answer sheet 
is read direct 1^ i^J^o ^he computer.) Based on our own experience, we would 
recommend a return to the original format, where forms were keypunched and 
the cards read into the computer. We say this for two reasons* First, the 
hardware fteeded to use mark-sense cards does not seem, as yet, to be 
sufficiently accessible, versatile, or even understood, coijlpared to more 
traditional data-entry systems. This generates problem^ particularly if onp 

♦ A ' / 
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wants to do more with the data than simply report aggregate statistics* (It 
might be desirable, for example, to link CRS d^ata to clients' R-300 - 
records. ♦ We have been unable, in an admittedly somewhat limited search, to 
. find_ anyone capable of doing this or other operations directly from mark- 
sense cards.) Until the needed technolo^ becomes more accessible, we feel- 
the old method is better, despite the added step (i.e., keypunching) which 
it requires. 

A second reason for switching forjnats is that, in our opinion, the 
current process of coding informatibn is more complex, and iavites mare 
coding error than the old process. It is harder for a reviewer to l03e his 
place when he records answers once, directly on the form, than when he has 
to transfer the information ^^-^^thre mark-sense card* 

Turning to the issue of the specific questions and their formats, the 
'following comments were made: 

"Better semantics (needed) frequently found terminology confusing." 

r ' 

"Phraseology 'confusing,, wordy and unclear.*' 
"Too many questions; many are repetitive." 

"Clarification needed to assure consistency among (reviewers)." 
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"Decisions are often based on %>inion of reviewer; opinions 
vary on interpretation." 

a 

"Forced answer choices often difficult to employ on inapplicable." 

In response to the first "fhree comments, we feel that the problems 
are overstated. The CRS questions actually follow a fairly easy-to-understand 
format; problems along this line should decrease as familiarity with the 
instrument increases. Further „^ our opinion the questions are not repetitive, 
although they do sometimes request increasingly detailed^nformation on the 
sami general topic • ' 

* The second three comments present more of a problem* - Some questions 
arc subjective; that is, they .require interpi;etation and judgment. ^ Other 
questions may require arbitrary judgments. However, we feel that these 
problems can also be mitigated by adequate training, particularly with respect 
to: 1) policy requirements ; 2 J acceptable/appropriate documentation; a^id 3) 
current best practice. We deal Avith these, issues further in our discussions 

*k . ■ ' 
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of necessary reviewer qualifications and training, needs; and in our 
recommendations for system implementation. • * 

Conflicts With State' Pplicy 

One of the major issues raised by reviewers concerned "their ^perception 
that the CRS irfcorrectly implies a unifoimity of policy and procedure across 
all states. Their range of concerns are conveyed best by the reviewers* own 
comments: 

• ''Before anybody tries to collect information at the state level, 
they should first check to see if the. policy and procedures at 

the federal and state level are the same, otherwise the data obtained 
could be unreliable or invalid." 

• "Each state has own policies and how can states be compared 
with one another?" 

• '^lodif ications ©eeded to cover peculiarities of (blind agencies. ) " 

• "Questions need more clarification. States have many options 

for, services and questions don't fit all molds. " (Emphasis added). 

• "Too much inclusion of .federal n:erms unnecessarily contained in 
CRS — should be a separate document. ^' 

Presumably, the prpblem here pertains more to case recording practices 
than to specific casework policies. After all, the law is the law and states 
, must comply with it. However, it would be unfair if, due to recording 
practices, a state seemed to be' out of qompliance when in fact it was not. 
Because the reviewers did not provide specific examples of problem areas, 
it is not possible to determine the importance of this issue or to suggest 
solutions. If the problem does lie in the realm of case recording, a possi- 
ble solution would be for the state and RSA to establish the existence of 
the problem beforehand, arid to -account for it when interpreting data results. 

Reviewer Qualificatiorrs and Training Requirements 

From an "implementation standpoint, some of the most significant comments 
pertained to reviewer qualifications and to training and process strategies. 
Because these issues are central to effective implementation of the full 
•procedural Standards 'process (i»e., R-300 Verifications and Timeliness 
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Assessments, as well as the Case Review Schedule), we discuss our responses 
and reooiiunendations in Chapter XT of this report, on iinplementat^on design.' 
Here, we simply present the comments, most of which we feel present valid 
points bleeding an 'appropriate responses 

• In order to achi'eve most reliable results, individuals whp - 
were more familiar with actual ca^wbrk procedures should have 
been utilized as reviewers* - • * ■ . 

'# ,CRS requires continual trainiTig to get consistency. 

User apathy was a problem, ' • ' 

^ -t Reviewer attentiorf was a problem, ^ 

• Introduction does not explain the purpose of the activity-. 

> • Siides helped considerably. " * * 

* * 

t IVould be helpful if R-300 verification preceded CRS trainijag. 

• .Might be nice if group could discuss some of mutually reviewed cases . 

after th^ fact. * * ' : 

/ i . _ 

• Too long. Better: many more reviewers with fewer cases per reviewer. 

One point /we should note here is that many negative comments were 
made about the/ training process employed during the pretest. Some of the 
pr^bJ^ms probaply resulted from the multiple purposes of and actors in the 1; 
pretesV, and would be les5 severe in on-going operatioil of the system. 



Pfesent^tion and Use of CRS Information ^ ^ ^ 

^ 'The fyrtal set of issues brought up by reviewers concerned the possi- 
bilities '^or using CRS data. Essentially, some r^ieweijrs . Were worried that 
the instrument would be used for administrative review only, with no , 
potenti^^l for use by line supervisors and with no feedback given to 'counselors. 
It is true that the broad purpose of the CRS, in the context of the Procedural 
Standards, is to allow states and, RSA go get an overall view of coiftpliance 
with regulations. However, in our opinion there is no reason why CRS 
information could not be disaggregated by counselor, and training targeted 
on that basis. In fact, the reviewer commerits* noted earler under' 
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"Specific benefits derived from th^ CRS" affirm this, assertion. The poten- 
tial for feedback is limited only by the staters willingnes.s to provide it 
(except, perhaps, for those agencies lacking EdV capacij:/^, for Jrtrhom provision 
of special assistance may be required}. 

Final iy,* some reviewers stated a preference for presenting the results 
,via percentages (i,e,., in terms of numbers) rather than via histograms. We 
agree.'^ Histograms* are somewhat hard to rea^, from' d precision standpoint; 
they require excessive s^ate; and they require top much effort to produce, par- 
^^^ularXy if distri^ct- level or counselor-^ t5vel data is desired. Of course, the 
exact method is the*' option of the reporting body; we simply prefei? the numbers. 

Summary • - "* * ' ' 
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Overall, the Case Review Schedule is seen as a good instrument for its 
intended purposes, and one^with a variety of quality-related benefits. Hqwever 
.it was al^o seen as requiring ex!cessive effort. Some t:oncems were expressed 
?egarJing the instrument-* s focus and form&t; we feel tKes^ concerns are mis- 
di reefed^' and ue feel that we hay§ ardeqtiately .explained our reasoning (see. 
pp. 28-50(1,; Overall, BPA^'feeLs that the CRS (or Modified CRS) would be 
appropriate for implementation as the data collection instrument for Stand- 
ards 10 and 12. However, prior to implementation, certain broad questions, 
must. b*e answered-.'. Th'ese are: ^ 

X \ \ 

♦ IVho will *hav^ responsibility 'for data collection? 

f How often yjp\l data collection .occur? • • ' • 

• How will cpnflicts between the CRS and ^*tates' policies be 
reconciled? ' ^ , j - ^ . - ' 

, IVhat qualifications must reviewjL^s have?- . ' % 

^# IVha.ftype of reviewer training is required? 



• . .How can the review process be made mpsr efficient *(ind effective? 



Because these issues concern all three Procedural "Starfdard^ data / 
collection instruments, we will address them later witii other issues 'cone err 
system de'sign>.. We thvn 7i5xt to the CR^S results obtain^d^(du]:ing the pretest, 



Results of Pretest*-Data 

In this sectioTT.we pre^-ent the performance levels and consistency 
ratios obtained from the pretest samples* The analysis is based on 
already-processed data provided to BPA by the San Diego RCEP. The discussion 
is organized by successive sections of the CRS, We begin with the data 
relevant to Standard lO. As noted earlier, our MEU revieivers generally 
approved of hoth the intents of Standards 10 and 12, and^of the particular 
CRS items used to assess performance ort the Standards. 

STANDARD 10: ELIGIBILITY 

— ^ ... 

Section IIA: Evaluation of ^Rehabilitation Potential; Preliminary 
Diagnostic Study - Status 02 

This sefction assesses the exteat to which the case record documents 
the occurrence of the various activities needed to conduct an effective 
preliminary 4iagpos^ic' study . This study, completed during the application, 
phase, should contain all of the information necessary to make a reasonable 
assessment of a client's eligibility for VR services. Among these necessary 
pieces' of information are all relevant medical and psychiatric examinations, 
and other evidence that supports the client's need and eligibility ^or 
rehabilita'^ion services. Without this informatioiyT' agencies will not be able 
to select the disabled individuals who can most benefit from available 
but limited VR services'. , ' ^ ■ ' 

Compliance with the regulations addressed in ^iSection llA was generally^ 
very good. In most cases, the information was reported 90% to 100% of the 
time. (See Table \0) . These figures can be considered reliable since the • 
corresponding consistency ratios were generally over .900. The only problema 
tic area vvas that of documenting the necessity for extended evaluation. 
(Items 6b and 11 in Table 40); on thesQ items, some performance levels fell 
belQw 50^. Considering thai: entry into extended evaluation- causes a delay 
in the full pr^vision of seWices, it is very imj5c)rtant for both the ag*ency 
and the client to know the' basis for that decision. ^ - . 

■ - ■ . ■ - ■ 

' ' . . . *■ > 

' . , ■ '- ' 

■ ^ ■■ . 

• . ' ' ■. ■ '281 ■ ' 
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Table 40 

Sfction II: Kvaliution of Rehabilitation Potential 

A: Preliminary Diagn^ostic Study - Status 02 



D01:S*Tlli: PKIiLIMINARY 
DIACNOSTIC STUDY, . . 






MS 


OR. 


PA 


VA 


4 - Include current 
health appraisal? 


Performance (%) 


100 


95 


< 99 


93- 


96 - 


99 


Consistency Ratio 


.981 


.955 


- ,1.000 


.983 


. .983 


.972 


5 - Include psychia- 
tric examinations if 
appropriate? 


Performance (%) 


100 


82 


63 


86 . 


.100 


98 


Consistency Ratio 


.933 


.742 


.917 


.875 


.917 


.875 


ba - Include eligibil- 
ity studios? V 


''Performance (%) 


«98 


■ 91 


99 


89 


98 


97' 


• 

Consistency" Ratio 


.1.000- 


.909 


.972 


.983 


• 1.000 


.931 • 


5b - Include need .for 
ex'tendcd evaluation? 


Performance (■s) ' 


100 


44 , 


100 


100 


_ 100 


67 • 


Consistency Ratio 


NA . ■ 


.886 


NA ■ 


.917 


NA 


.917 


7 - Place emphasis 
on achieving voca- 
tional goal? 


Performance (%). 


93 


91 


92 


93 


98 


97 


Consistency^ Ratio ' 


.981 


.848 


.958 


.950 


.983 


.931 


8 - Support determi- 
nation of disability? 


Performance (%) 


' 98 


97 


100 ' 


93 


100 


99 


Consistency Ratio 


1 .000 


.955 


.986 


.983 


.967 


.931 



( '2S2 . _ • " . . 2S3 

O . . . ' ■ - * 
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CRS QUESTION- WORDINGS . 

The exact wording of the Case Review items for Section IIA, per- 
taining to Standard 10, appears^ below. 

Does the preliminary diagnostic study... * ' » ' 

4. include an' appraisal of the current general health status 
of the client? 



\ 



5. include a psychiatiiic or psychological examination in all 
^ cases of mental or emotional disorder? ^ 

6. include such examirtations and diagnostic studies as necessary 

to: ^ ^ . ^ 

a. determine eligibility? f 
* * b. determine /the need for extended evaluation? 




7. *place primary emphasrSsjopon^&tie determination of the client's 
potential for achi^eving a vocational goal? 

8. support the 4etermination that the client has a medically- 
recognized physical or mental disability? 



Table 40 (.cont.) 



DOi:S Till: PKHLIMINARY 
DTAGNOSTIC STlinY.. . ' 


ni- 


Ml 


MS 


OR 


PA 


VA 


9 - 'Support disabilit) 
• as- handicap to cm- 
' ployment? 


Performance (%) 


97 


98 


97 


87 


• 97 


■ 97 


-insistency Ratio 


.981 


T — : 

.909 . 


.944 


1.000 


.933 


.944 ■ 


10 - Support, that 


Performance (%) 


90 


92. 


95 


. 87 


98 


94 , 


' services will bene- 
fit client? 


Consistency Ratio., 


.' .981 


.924 • 


;889 


-^4 

.983 


.967 


.903 - 


11 - Support that 
-'^F.xtentlcd Lvaluation 
is necessary? 


Performance (%) 


• 100 


36 


93 


100 


100 


lop 


Consistency Ratio . 


.889 


.818- 


.889 


' NA 


. NA 


.889 


• N = 


• 120 • 


. 119 


119 


100 


— 

98 


118 


• > 
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CRS QUESTION WORDINGS 



. The exact wording of the Case Review items for Section IIA, per 
taining to Standard 10, appears, below. 



Does the preliminary diagnostic study... 

9.* support the determination that the medically-recognized 
disability constitutes a substantial handicap to employ- 
ment for the client? 

10. support tYie determination that VR services may reasonably 

be expected to benefit the client in terms of employability? 

11. support the determination that an extended evaluation is 
necessary to determine that VR services may reasonably 

be expected Jto benefit the client in terms of employability? 
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• Section IIB: 'Valuation of Rehabilitation- Potential; Extended Evalu ation i 
St^us 06 ' \ = = 

SectAjn IIB seeks documen^tion tJ^t the state agency has followed 
^pjroper procedure in placing applicants into extended evaluation,- status (?6. 
For tjie Procedural Standards, our ^concerns are that case records include: 

• • certification for extended evaluation to determine rehabilitation 

potential;. 

• the 5asi5 for the need for extended evaluation; 

^ « 

• evidence of the occurrence of thorough assessments of progress at 
least every 90 days; and, 

• documentation of the eligibility decision resulting- from extended 
evaluation, , ^ ' ' 

Provision of this information helps ensure that exte.rided evaluation is used 
only when appropriat'e, that the client moves through extended evaluation in 
a timely manner, and that the' minimal recording needs for Status 06 are main 
tained. , ^ ^ 

Table 41 presents the results of Section MB. Several data problems ar 
in this section. Virginia has not been included in this table*, since that 
state apparently did not review any cases of clients placed into status 06. 
In other states, the low numb'er of extended evaluation cases included in the 

3iability sample precluded computation of. consistency ratios. Also, no 
a was available on 08 closures, thus results for items 42a and 42d,. per- 
taining to ineligibile applicants, are not available. 

In general, state compliance appeared fairly high in terms of: 

• including a certification for extended evaluation in, the case 

record (item 14) ; * ' 

/ • 

• presenting the basis for the need for extended evaluation (item 19); 
and , ' ^ 

. • • i 

• documenting that a certification of eligibility for continued 
services was completed, when appropriate (iteirf^40). 

However, states apprently less often document the occurrence of 90-'day ' 

■ , 2SS 
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rev^i^s required under status 06 (item^ 33) . It is important that the case 
record contains this information, or the agency will not be able to assess 
the quality and appropriateness of services^ given during extended evaluation. 

Section III; Eligibility - Status 10 

UTiile it is important to document in the preliminary diagnostic study 
the extent to which applicants meet the basic eligijjility criteria, it is 
evenjnore important to ensure that all clients accepted for services meet all 
of the requirements for eligibility. Trfe purpose of Section III is to document 
that a certification of eligibility was completed for each accepted client, and 
that\ counselor .documentation in the case record confirms: 

• the' existence of a disability; ^ ^ - . ^ ' - • 

• the existence of a .substantial handicap to employfhent; and 

• the likelihood that VR services will benefit clients. 

t 

The results of the pretest (see Table 42) show that at least 98% of ^ 
the certifications of eligibility indicate that clients met these three 
requiretnents (see item IC) . ^ The next three items seek to document that counselor 
records corroborate the information contained in the certification of eligibility 
State performance was generally good in terms of .documenting the presence 
of a cTisability ; -this occurred in at least 79% of ^ach state's cases (see 
item 3). However, the remaining *two eligibility criteria proved more problem- 
atic. This can be seen by scanning down each individual state's column in 
Table 42, and comparing the results obtained on Item 3 to the results obtained 
on items 7 (concerning documentation of an employment ^andicap) and 8 .(docu- 
menting ^treasonable expectation'*). Note that ipn items 7 and 8, each state | 
registers lower performance levels and lower con4istence ratios that it 
obtained on item 3. Stated operationally, these findings imply: 

I. that states are less diligent in documenting the existence of 
A employment handicaps and ^'reasonable expectations" tlian they are ^ 
in documenting existence of a disability (as shovm by the lower 
performance -figures) ; and 
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Table 41 

Section JIB: livaluation of Rehabilitation Potential; lixtendcd Evaluation '^tatus 06 
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1 

nOliS- Till: CASI-; RECORD/.. 


DE 


Ml ^ 


V MS 


OR 


' PA 


14 - Contain certifi- 
cation for extended 
, evaluation? . 


Performance [%) 


73 


0 


V 

100 


100 ' 


b 


Consistency Ratio 


.889 


.909 


NA 


NA 


.917 


IXM-S Till. WWV . , . ^ 


• 100 










19 - Present basis 
for need for extended 
evaluation? 


Performance (%) 


88 


100 


-- 


Consistency Ratio 


NA 


NA 


NA 


:' NA \ 


NA 


33 - ShQw that' re- 
view of progress was 
mtide under extended 
cvulurrt ion? 


Performance (%) 


78 


-- 


29 


100 


-- 


Consistency Ratio 


■ NA 


NA 


NA 


NA^ 


NA 


fJpi.S lin. CVSL RIXOKD, , , 




- 


88 • 


• 




40 - Ccjntain a cer- 
tifi-cation of eligi- 
bility for continued.' 
services? 


Performance f%) / 


100 




Consisj:ency Ratio 


NA 


NA 


NA 


NA 


f NA 


42a - Show decision 
to terminate was 
made with consulta- * 
tion?' ' • 


Pei'formance (%j 












Consistency Ratio 


NA' 


NA 


' NA 


NA 


\ NA 


42d - Show thatT pro- 
vis ion was made f^r 
review? 


Performance f%) 












Consistency Ratio 


NA 


NA 


NA 


NA 


NA 


N = 


r20' 


119 


119 


100 


98 
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CRS QUESTION WORDINGS. 



The exact wo'rding of the Case Review items for Section HB, per 
taining to Standard 10, appears , below. 



Does the case record... 



14, contain a certification for extended evaluation tp 
deterfhine rehabilitation potential? 



Does the IWRP for extended evaluation (state fbrm) . . . 

19. present the general basis for a deterijiination that an 
extended evaluation of rehabilitation potential is nec- 
essary to make a determination of eligibility? 

33, show that a thorough assessment of the client's progress 
was made at least once in every 90-day period during the 
provision of services under the extended evaluation? 



Does the case record... 

40. cdntain a certification of eligib|,lity for the continuance 
of VR services? 

42a. show that the decision to terminate services*^ was made in 
full, consultation with the client, or as appropriat*e, with 
the parent, guardian, or other representative? 

42d. show ^hat tlie provision was made for a periodic review, at 
least annually, of the ineligibility decision? 



V 
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Section III:^ Hligibility - Status 10 



DOES THIi Cnrrri'ICATION Ol' jiligibility,.. 


DE 


MI 


MS 


OR 


PA j 


VA 


Ic - Indicate client 
has met eligibility 
requirements? ' 


Performance (%) 


100 ' 


98 


98. 


99 


99 ^ 


98 


Consistency Ratio 


1.000 


.955 


.958 ' 


i.ooa/. " 


.933 . 


. .944 


now WELL DOES THE COUNSELOR 
DOCUMENTATION IN TIIH CASE RECORD... 












3 - Establish ,the 
presence. of a dis- 
ability.? 


Performance (%) 


9v7 


79. 


98 


88 


93 


89 ^ 


Consistency Ratio 


.981 


.742 


.972 , 


.983 


.917 ' 


.819 


7 - Show that handi- 
cap to employment 
exists? 


Performance (%) 


81 


65 


w 

88 


62 


86^ 


77 


Consistency Ratio 


.778 


ar 

.682 


.653 


.917 


.617 


.694 


8 - Show that ser- 
vices xan help client 
achieve godls? / 


Performance (%)• 


80 


64 


79 


8-4 


91 


81 


Consistency Ratio 


.889 


• ..652 


.792 


.833 


.883 


.833 


N ^ 


120 


' 119 


119 


100 


' 98 


118 



i 
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CRS QUESTION WORDINGS ^ . 
The exact wording the Case Review items for Section III^ pe 



taining to Standard 10, appears below. 



Does the certification o*f eligibility.., 

Ic^. indicate that the client has met the basic eligibility 
requirements? ^ 



How well does the counselor documentation in the case record,,, 

5. establish the presence of a physical or mental disability 
with necessary medical, psychiatric, psychological*, and 
other information? ' i 

show that the substantial handicap to employment exists, 
. ^ ^ even though the client is employed., because the client 

is unable to obtain a gainful occupation consistent with 
the client's capacities and abilities? 

S, show the likelihood of VR services enabling the client' to 
- achieve' vocational goals consistent with the client's 
c;ipacities and abilities? 
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*2. that it is harder to establish whether documentation exists or 
not (as shown by the lower consistency ratios). * 

Both of these, findings may stem from the more subjective and * judgmental 
nature of the latteA^wo criteria, in relation to the fir^t criterion. I't 
should a fairly straight-forward task to establish the existence of a 
disability. This requires no more than a doctor's certification that a 
person is blind -(^or paralyzed, or deaf, or whatever). Functional assessment 
results would equally suffice to establisli the presence of a disability. 
Further, a doctor's note, or test results would not ju'st document presence 
of a disability/they woul-d document it unambiguously . That is^ all reviewers 
would presumably agree that such things constituted adequate documentation .j 
If this argument is true, then we would, expect both a high performance level 
and a high consistency ratio on item 3. 

In contrast, items 7 and 8 require more judgment both of the counselor 
(or other intake/eligibility worker) and of the case reviewer. From the 
counselor's vie\^oint , ^any ot^er considerations must be accounted for when 
deciding whether a handicap to employment exists. (For example, does the 
disability preclude any employment, ^r just the kinds of employment the ^ 
client vvould prefer? Is the disability severe enough to truly preclude 
employment, or could the client perhaps get a job even without VR help? 
And ^o on.) Given the potential complexity of such a judgment, the counselor 
may forego rigorous consideration of this issue, and instead rely on intuition 
afid past experience. To the extent this occurs, counselors are/less Ukely 
to Jocu::ient formally the existence of an employment handicap. Turning to 
the reviewers, the complexity of the issues involved in such a judgment will,*- 
almost by definition, make it more difficult to judge ivhether adequate 
documentation exists. Further, since reviewers are charged with judging 
the quality of counselor decisions as well as* judging the extent of qounselor 
documentation, the potential for ''prof essional disagreenient'^ increases . ' That 
is, one reviewer may judge th6 documentation provided by the counselor as 
adequate to establish an employment handicap, whije another reviewer judges.it 
inadequate. In short, given the complexity of the issues, we could reasonably 
expect lower consistency ratios. These problems, for counselors .and revi'^wers, 

\ ' ' ' 
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are heightened further when attempting to establish "reasonable expectation 
IVhile explanations can be offered for the iQwer ratings found on items 
7 and 8, we would still consider* the ratings as problems needing correction. 
In oui^opinion, documentation should be such that there is no doubt that 
clients have me^ the basic requirements for eligibility^ 

Section VII: Teumination of Cases 
— ~ ^ — I — 

Section VII of the full case Review Schedule contains questions 
pertaining to the termination process for all closure statuses • The Procedural 
Standards, in contrast, focus on non-successful closures: OS's (from both 
02 and 06), 28's and 30's* For these closure statuses. Standard 10 attempts 
to assess, (through review of case record documentation) the following 
compliance issues: •% 

• Does the case record document the ineligibility/termination decision, 
^ and the basis for that decision? 

Have clients been granted their legal rights to participate in the 
ineligibility/termination decision? • ^ 

• Haye clients been inforijied of their right to an annual review of the 
decision? and, 

• Have the required annual reviews occurred, and the results documented? 

The need for this information is two-fold. First, it is reasonable to 
expect that supervisory personnel might want to review any given case iir/olving 
ineligibility or unsuccessful termination for any given counselor. Manage- 
'ment personnel should have the ability to review cases sampled at random 
(i.e., across all counselors, as in an audit- type procedure), or to target 3^ 
reviews to. particular counselors (as might be needed for less experienced 
counselors). Either way, it follows that for any case of ineligibility or 
unsi^ccessful closure, the closure action and the basis for it must IJe ade- 
quately documented in the case recoi:d. [ 

'.This information in Section VII is important for second reason, 
which stems from VR's desire to protect the rights of its applicants and 
clients. The best way to ensure such client protection is to require proof 
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in the' case record that the necessary steps have occurred: for example, a 
"Bill of Rights" signed by the client; a schedule for review, signed periaps 
by the client; and "Results of Review" form, which could be signed by the 
client. ^ 

In short, VR agencies need to know the reaons for unsuccess^ful cloWes, 
and need fo ensure that ineligibles and unsuccessful closures are aware 
of their rights to review. Once this is ensured, then informed clients — 
whose circumstances have changed such that they are eligibile — hopefully 
*will re-entei^-'^tRe system later and are successfully rehabilitated. 

Table 43 presents the Section VII Results obtained in the pretest. Due 
to the absence of 08 closures in the states' samples, we. are unable to^ test 
the CRS items pertaining to ineligibility determinations. However, so that 
the reader may understand the thrust of this standard with respect to 
ineligibles, we present the relevant question wordings in Figure 5. We 
feel it is most unfortunate that 08 closures were excluded from the sample. 
In effect, this exclusion means that we can assess compliance only for one 
of 1 the two "sides" of the eligibility question:- that is, we can address 
th* issue of whether or not .clients admitted for service should in fact have 
been admitted (the eligible "side"); but we cannot assess whether those • 
denied services should, on the basis of documen<:4tion, in fact have been . 
denied. At any rate, the data collection is a fait accompli. The following 
data discussion will be limited to the issues of 28 and* 30 closures. 

As can be seen in Table 43, we obtai^i variation in performance levels 
and cor^ist^cy ratios for this ^'sfect ion. , The table indicates that in many cases 
there is not unambiguous documentation that the client was consulted about 
the ^lostire decision, or was informed of the right to participate in an 
annual review, nor was there' documentation that the decision was revie>ved 
within one .year (see items 35, 37b, 59). The table shows that many clients 
are not involved in the closure process, possibly due to loss of contact 
between clients and the agency. In any case, agencies must be sure they 
inform as many clients as possible of their rights as status 28/50 closures, 
and that they provide the required annual review. 
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Figure 5 

SECTION VII. ITLMS PERTAINING TO 08 CLOSURES: NO DATA AVAILABLE IN* PRETEST SAMPLE. (-Numbers in parenthesis 
indicate Case Revievv item numbers J ~ 

CASS CLOSED STATUS 08> FROM 00/02 - INTERVENING REASONS 

4 

Does the case record... 
(2)' / document specific reasons for the closure action? 

(S) show that the client, or as appropriate, the, parent, guardian, or other representative, was advised 

of the reasons for closure and the closure action taken? 

CASE CLOSED STATUS 08 FROM .00/02 - INELIGIBILITY . ^ 

Does the certification of inel igibilit);^.. * ' ^ 

(6a) indicate the date of certification? 

i6c} include the reasons for the determination of ineligibility? 

Does the case record.., 

(7) show that the client does not have a medically recognized physical or mental disability? 

(8) show that the client does not have a substantial handicap to employment? 

(9) show that beyond any reasonable dout)t the client is not expected to benefit m terras of employability 
from VR services? * 

contain data supporting the ineligibility determination, including; 

yOa) a surxiary of medicai and other case data obtained during the preliminary diagnostic study? 

(lOd) documentation, of a review of the ineligibility determination not later than 12 months following 

such determination 

(11) show that the ineligibility determination was nvade only after full consultation with the client, or as 
appropriate, with the parent, guardian, or other representative? 

(12) document that the client was notified in writing of the closure action taken? 

document that the client was informed in writing of client rights and remedies,* including: " 
(13a) , the right to adfhinisrative review and fair hearing? 

(13c) the right to participate in the annual review of the ineligibility determinatiort , 

(14) document any action and decision involving the client *s request for an administrative review of 
- agency action or fair hearing? 

« 

CASE CLOSED STATUS 08 FROM 06 > INELIGIBILITY 

Does the certification of ineligibility... 
(17a) indicate the date of certification? . 

(17c), include the reasons for the determination of ineligibility? 

Does the case record... 

(15) . show that beyond any reasonable doubt the client cannot be expected to benefit in terms of employability 

from VR services? 

(20) 'contain the rationale for the ineligibility determination as an amendment to the lU'RP? 

(21) sho^ that the ineligibility determination uas made only after full consu.ltatio^ uith the client, or as 
appropriate, with the parent, guardian, qr, other representative? 

(23) document that the client was informed in writing of^ the closure action taken? ' 

(24) docunent any action and decision involving the client's request £pr an administrative review of 
f agency action or fair hearing? 

(25) document that the. ineligibility determination was reviewed not later than 12; months following such 
determination? ^ 

^ * * f 
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Section VII: Tcnninatdon of Cases ^Statuses 30 aW 28 



DOIiS THl:-CCRTll-ICATrtw W 
•INELIGIBILITY. . . ' ' ; . ' ' 

« . . :^-4 : 


Dt 


MI • ' 


. MS. ' 


- OR 


i'A 


' .VA . 


A / , 

( 

w 

2&a - IncTicate date 
• of incHgibi lity? 


» 

. Performance (%) , 


■100 


93' ■ 




- 1^0 , • 


6S , 


■98 


Consistency Ratio. 


NA 


'■ ;576 ' 


NA ' 


■ *t 

. 1.000 


^- .833- 


* 

. l.ffOO 


28c - Include refisons 
- for ineligibility? 


Performance ;(%) - 


IQO 


i 92 




•98 • 


• ■ 65 


. 98 


Consistency ^Ratio^ 


* NA 


- .606 


NA 

J 


1.000 • 


.792 


, ■ .'9.72 • 


. DOES THE fcASE .RECDRLfT 












34 - Contaij^ ration- 
ale^ as an amejpdinent , 
to IWRP : ^ 


Performance 


92 • 


70 ■ 


%. 

71 


/ 100 


' 83: 


• . 98 


Consistency Ratio 


.815- 


^ .455 


.583 • 


. 972 


'..833' 


.917 


35 - Show th^t deter- 
milnatiori was made ^ 
after consultation? 


Performance (%) 


* 78 


. 59 . 


sa 


70 


. . 77 


74 


Consistency Ratio 


.6^0 


.485 


.722 


\ 

.722 


.817 




36 - Document that • 
client was notified 
in writing? 


Performance (%) 


36 , 


•89 


f 61 ) 


86 


} 

100 • 




Consistency Ratio 


.778 


.875 


.-722 


.889 


.933 


.861 


37a - Show that clieni 


Performance (%) 


' 93 " 


^ 63 . 


56 . 


,89 


I- 

88 


96 


was informed of right 
, of review? 


Consistency Ratio 


-.778 


.606 


.556 


' ,944 •• 


.817 


. 833 
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CRS QUESTION WORDINGS 

The exact wording of the. Case Review items for Section VII, per- 
taining to'Standard 10, appears. below. ^. ^ 

.Does the certification of ineligibility... 

28a* indicate the date of cer'&if Kation? 

28c. include the reasons for the determination of ineligibility? 

Does the ca§e record... ' ' 

34. contain- the rationale for the ineligibility^ determination 
as an amendment to the program? ^ 

35. show that the ineligibility determination was made only 
after full consultation with the client, of as appropriate, 

* wj.th the parent, guardian, o*r other representative? 

* 36. document that the client was notified in writing of the, 
» closure action tgtken? 

4 

Does the case record show that the client was informed in writing of.. 

r 

37a. the right to administrative review and fair hearing? 
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Tabic 4'3 (cont.) 



^VOES Tim CASH RhCORD'. . . 


DE • 


MI 


^ T- 

'ms 


OR 


PA 


VA 


37b - Show that 
client was informed 
of rrght *to partici- 
pate in annual review 


Performance (%) 


91 


2 




20 


SO 


96 


* Consistency Ratio 


.741 


.758 


. 9S6 


.667 


.717 


.833 


38 - Document act-ion 
^involving request 
forrfftjview,? 




9 


31 




* 

13 


50 


100 


Cqf^gistency Ratio 


. .889 


S 

.758 


.8^ 


.861 


."883 


.722 - 


, 39, -document that 
detQrmination was 
•reviewed within 12, 
mo'hths? 


•Performance (%) 


0 


1-3 


50 . 


0 


43 


* 

71 " 


Consi'stency Ratio 


.889 


.788 


.875 


- ,833 


.917 


,778 




N = 


-> 

120 


,"ll9' 


11^ . 


100 


98 


118 



to 

CO 

to 
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. CRS QUESTION WORDINGS 



The exact wording of the Case^ Review items for Section /-VII, per-^ 
taining to Standard 10, appears below. 



Does the case record show that the client was infexmed in v^iting of. 

37b. the right to participate in the annual review of 
the ineligibility determination? 



Does the case record... 

38. document any action and decision involving the client's 
request for an administrative review of agepey action 
or fair hearing?/ * ^ 

59. document that the determination that the client was no 
• rJ5nger eligible w'as reviewed not later than 12 months 
following such (determination? 
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STANDARD 12: IWRP 



Section IIB: Evaluation T>f R^^^ilitation Potential; Extended Evaluation - 
Status 06, 

This section seeks ^o dopument that the IMF's for clients placed into 
extended evaluation cqntain all of th'e information required under status 06. 
In this context, tlie concerns of the Prc^pedural Standards are that IV/RP's 
for cases entering 06 do the following: 

define the terms and conditions for the provision of services; 

• -document that -the client was informed of specific rights, including 
.«r--^ the right to participate in the development^ of the program; 

• specify vocational goal and a timeframe for its achievement;* 

• specify evaluation procedures and criteria; 

• documbnt the final eligibility decision and', for those clients 
closed as ineligible:^ 

document that proper information recording procedures were 
\ followed; 

-•; document that the client participated in the decision; and 
— document that provision was mad^ for periodic/^review. 

Provision oj^ Jthis information helps ensure that the extended evaluation 
status is used only when appropriate, that clients move through status 06 
in a timely manner, and that clients are aware of their rights to continued 
services or r'eView if declared ineligible. 

Table 44 presents the results of Section II-B for Standard 12. As ; 
was the case for Standard 10, we encountered data' availability problems here. 
Michigan, Pennsylvania, and Virginia do not appear in the table^^^cause 
their samples apparently included no cases entered status 06. Further, the 
remainirig* three states' app^^icable samples (i.e., clients who entered, 06) 
were very small. F^lr example, Oregon's 33% performance rating on item 30 
is based on only three clients. Given these' small numbers, and the inability 
to compute consistency i;ations|f caution should be exercised when interpreting 
the results. 
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1^ = ^ Table ^ r 

Section IIB: Evaluation of Rehabilitation Potential ; Extended Evaluation - Status 06 



DOES THE CASE RECORD... 


.DC 


MI 


MS 


OR 


PA 


f 

VA 


r 

f 

18 - Contain an 
IWRP for Extended 
'Evaluation? 


Performance (%) 


83 . 




70 


75 




i 


DOES THE IWRP,,, ^ 














19 - Present need 
for Exteifded Evalua- 
tion? 


Performance (%) 


100 


• 


88 


100 * 






6 

20 - Describe client 
responsibilities? 


Performance f%) ' 


* 90 ■ 


■ V 


75 


100 






20b - Describe client 
participatJWi? \ 


1 

Performance ■ f%) 


» 100 


• t 


75 - 


100 






21a - pocujnofit 
client's right to be/ 
fully consulted? I 


Performance (%) 


100 , 




75 


'67 


^ 




\ ^ ^ 

21b - Describe the 
client's right to 
•review?* 


Performance f%) 


100 




63 


100 

— — 1 I 


J 


"I 
* • 



Michigan, Pennsylvania,* and Virginia are excluded from t. 
no ;data for this «sT3ction. 

ERIC 
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CRS QUESTION WORDINGS 

The exact wording of the Case Review items for Section I^IB, per- 
tairling to Standard 12, appears below. 
/ 

18. Is. there a? IWRP for extended evaluation in the case recof<^ 
> 

Does the IWRP for extended evaluation (s|ate form)... ^ 

19. present the' general basis for a determination.. that an« 
extended evaluation of rehabilitation potential is 
necessary to make a determination of eligibility? 



20. 



Set forth the terms .and conditions for the 'provision of 
service, including: 

a. client respoirsibilities in carrying out the 

program, such as attendance, cooperation, ^etc? 

b; the extent of clientVparticipation in the cost 
of seryfices? 



21. document that the client was informed of client rights and 
remedies, including: 

af. the right to be fully consulted regarding"^ 

any changes t)r amendmenty^in tKe rehabilitation 
program? ^ 

b. the right to administrative review, in case of 
dissatisfaction with services? 
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Tabic 44 (coiit:) 



r 



DbES TliO IWRpA . 



DP. 



MI 



MS 



OR 



PA 



VA 



21d - Describe the 
client's right to 
participate in review 



Peftformance (%) 



100 



50 



67 ' 



21e - Describe the 
client's right to 
parficipate in re- 
view of ineligibility 



Performance* (%) 



' lOO 



63 



67 



22 - Show the IWRP 
as a separate part? 



Performance (%) 



100 



75 



100 



23 - Show client 
received IWRP? 



Performance (%) 



100 



88 



100 



26 - Indicate client 
participation in 
developing program? 



Performance (%) 



100 



88. 



100 



28 - State interme- 
diate rehabilitation 
objectives? 



Performance (%) 



100 



50 
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CRS .QUESTION WORDINGS 

/ 

The exact wording of the Case Review items for Section I IB, per- 
taining to Standard 12, appears below. 

Doets the IIVRP for extended evaluation (state form)... 

d. the right to participate in the annual review of 
the program? 

e. the right to participate in the annual review of 
, J the ineligibility decision? 

22 ."C^ref lect that the IWRP for extended evaluation was maintained 
as a separate part of the case record?' 

23. show that the client received a copy of the IWRP and sub- 
stantial amendments? 

26. indicate that the program was developed and amended with 
the i:lient'^s participation, gr^ as appropriate, with the 
parent, guardian, or other representative? . • ^ 

28. staj;^'the intermediate rehabilitation objectives? 
/ 
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Table 44 (cont.) 




DOHS TliH IWRP.. . 


1)1- 




MS 


OR 


I'A ' 


VA 


29 - Contain neces- 
sary services? 


Performance [%) - 


100 




/ • 

63 


100 , 


• 




30 - Contain initia-v 
tion'date for ser- 
vices? 


Performance f%) 


100 




75. 


s 

33 






31 - Contain antici- 
pated duration of 
services? 


Performance f%)- 


100 




75 




< 




^ 

32 - Provide achieve- 
ment time for objec- 
tives? 


Perfbrmarice f%) 


^ — ' 

100 




63 


67 


V 




33 - Show an Extended 
Evaluation assess- 
ment every 90 days? 


Perform^^i^ C%) 


' 78 




29 


100 






1 

34 - State Evaluation 
criteria? 


Ci 

Performance ■ (%) 


. " 90- 




38 , 


100 
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CRS QUESTION WORDINGS ^ 

The exact wording of the Case Review items for Sectiorf IIB, pe: 
^taining to Standard 12, appears below. 

Does the IWRP for extended evaluation (staje form).., 

29, state the VR services to be provided which are necessary- 
for the determination of rehabilitation potential? 

30, contain the projected daj:e for the initiation of each 
servi ce? 

31, contain the anticipated duration for each service planned? 

'32. provide the projected time within , which rehabilitation 
* objectives may be achieved? 

33, show that a thorough assessmejit of the client's progress 
was made at 'least once in every 90-day period d,uring the 
provision of services under the extended evaluation? 

34. state the objective criteria upon which an evaluation of 
the client's progress is based? 
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Table 44 (cont.) 



DOES THE IWRP.../ 


DE 


MI 


MS 


OR 


PA 


VA 


35 - State evilluation 
criteria? / 




90 




25 ^ 


100 


s 




■ — 

36 - Cont^n schedule 
for revieiws and 
evaluation? 


Performan^ f%) 
Performance (%) 


60 




50 


100 






37 - ||?ecord results 
of reviews and 
cvajiiation? 


Performance f%) 


20 




25 


• 

100 






m - Show, annual 
nWRP review? 


Performance f%) 














39 - Document client 
opinions concerning 
objectives and ser- , 
vices? 


Performance (%) 


70 




75 


100 






DOES THE CASE RECORD,.. 














42a - Show termina- 
tion was made with 
client consultation? 


Performance (%)' 
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CRS QUESTION WORDINGS 



The exact wording of Case Review items for Section ^ilB, per- 
taining to Standard 12, appears below. , . ^ 



Does IWRP for extended evaluSti<2ii_C§JtM^ form)... 

client is eva 



35. state the procedure by^hich th 

36. contain a schedule foV the peri 
evaluation? 




)dic review and-^rpgres^ 
of scheduled reviews and 



37. contain a record of the results^ 
prog^res^ evaluati^s? 

38. ' show that a formal, annual review has been conducted if the 

IWRP ti^s achieved at least first anniversary status? 

* 39. document^e client's views, or-, as appropriate, the views 
of the parent, guardian, s>t other representative concerning 
the objectives and VR services being provided? 

Does the case record .>.. 

42a. show that the decision to terminate services was made in 
full cc^sultation with the client, or as appropriate, with 
the parent, guardian, or, other represent at iv$p? 



0 

) ■ 
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Table 44 (cont.) 


DOES THE CASE RECORD. . . 




MT 


MS 


OR 


PA 


VA 


42b - Show rationale 
as amendment to 
IWRP? 


Performance (%) 


— 




— 








42c - Show certifica- 
tion of ineligibility' 


* 

Performance (%) 






— 


— 






42d' - Show provision 
was m^defor annual 
review? * 


Performance (%) 


'-- 










• 


DOES TH£ CASE RECORD DOCUMENT THAT 
THE SERVICE WAS PLANNED... 












* 


43 - Diagnostic and 
related services? 


Performance (%) 


58 




. 90 


100 






44 - Counseling and 
guidance? 


Performance (%) 


50 




" 78 




« 




. \ 

45 - Physical restor- 
ation? 


Performance (%) 


33 




70 


50 






^ ' / .. 5i4 , '■ - 
ERJ.C ,^ . \^_^ . . . , , . - . , . 
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CRS QUESTION WORDINGS 

♦ 9 

* t 

The exact wording of the Case Review items for Section IIB, per- 
taining to Standard 12*, appears below ^ ' ^ 

t 

Does the case record... 

42b. show -that tl\e rationale for the decision ^o terminate 
services was recorded as a certified^ amendment to the 
IWRP for extended evaluation? 

42c. show that a certification of ineligibility was then executed? 

42d. show that the provision was made for a periodic review, at 
least annually, of the ineligibility decision? 

Item B: Does the case record document that the service was planned 
for the client? 

Item 6 is asked in reference to the following services (keyed to CRS 
question numbers) . 

Services : ' ' * ' . ' 

43. Diagnostic and Related Services 

44. Counseling and Guidance 

45. Physical Restoration 
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Table -lA (cont.) 



DOES THE CASE RECORD DOCUMENT THAT 
THE SERVICE WAS PLANNED.., 


DE 


- MI 


. US 


OR 


PA 


■ VA 


46 - Mental restora- 
tion? . 


Performance (%) 


25 








• 


• 


47 - Vocational and 
other training? 


Performance (%) 


50 




• 75 


25 






• 

48 - Maintenance? 


Performance (%) 


8 




44 


• 






49 - Transportation? 


Performance (%) 


42 




33 


50 






50 - Services to the 
family? 


^Performance C%) 


I 

r* — 


/ 










51 - Specialized 
services for blind, 
deaf, severe dis- 
abilities? 


Performance (%/ 


-- 




J* 

44 









4 



' : * ': 316 

ERIC 




297 



/ 



CRS QUESTION WORDINGS , 

The exact wording of the Case Review items for Section hb, per- 
taining to Standard 12, appears below. ^ 

« 

Item B: Does thp case record document that the service was planned 
for the client? 

Item B is asked in reference to the following services (keyed to CRS 
question numbers) . 

Services : 

46. Mental Restoration C 

47. Vocational and Other Training 

48. Maintenance 

49. Transporation 

50. Services to the Family 

r 

51. Specialized Services for Blind, Deaf, Severe Disabilities 
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Table 44 (cont.) 




« 




- 


4 


> 


DOliS CASli RliCORl) DOCDMIiNT THAT 
T!IO ShRVICH WAS PLANNHI)..". 


\)\: 


MI 


MS 


OR 


PA 


VA 


52 Telecommunica- 
tions? 


Performance (%) 


-- 




11 


1 

-- 






53 - Occupational 
licenses^, tools, 
equipment? 


* 

Performance (%) 


• 




1 

I 








54 - Other gopds 
and services^ 


/ ) 

Performance C%) 


42 ■ 




30 


— 


* 


% 


flOES THE CASE RECORD D0CUf4ENT THAT 
THE SERVICE WAS GIVEN.., 














43 - Diagnostic and 
related services?. 


Performance (%) 


58 , 




90 


100 






44 - Counseling and 
guidance? 


Performance (%) 


42 




100 


-- 






45 - Physical res tor - 
|ation? 


Performance (%) 


33 




> 

100' 


50 







'* Pennsylvania was able to computate item 54: Performance = 100, Consistency Ratio = .917 
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CRS QUESTION WORDINGS 

The exact wording "of the Case Review items for Section •HB, per- 
taining to Standard 12, appears below. 

Item B: Does the case record document that the service was planned 
for the client? 

Item B is asked in reference to the following services (keyed to CRS 
question numbers) * 

Services: 

52. Telecominunication's 

53. Occupational Licenses, Tools, Equipment 

54. Other Goods and Services 

Item C: Does -the case record document that the service wa^- given to 

the client? ' ' , 

* I • 

Item C is asked in reference to the following services Ckeyed to CRS 
question numbers) . 

Services: . ■ , 

I 

43. Diagnostic and Related Services ^ 

44. Counseling and Guidance 

45. Physical Restoration 
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Table 44 (cont.) 












DOES THE CASb ROCORD DOCUMENT TFiAT 
THE SERVICE WAS GIVEN... 


. DE 


MI 


MS 


OR 


PA 


VA 


46 - Mental restora- 
tion? 


Performance (%) 


17 




33 


- 


t 




sA7 - Vocational and 
ither training? 


Performance (%) 


42 




100 


25 


J 




48 - Maintenance? 


Performance (%) 


8 


* 


60 




) 




49 - Transportation? 


Performance f%) 


42 




, 75 


25 






50 - Services to the 
family? 


Performance (%) 








4 






51 - Specialized 
services for blind, 
deaf, seveire dis- 
abilities? 


Performance (%) 






80 
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CRS QUESTION WORDINGS 

The exact wording of the Case RevieW items for Section 'IIB, per- 
taining to Standard 12, appears below. 

Item C: Does the case record dociiraent that the service was given to 
the client? 

Item C is asked in reference to the fo Rowing services (keyed to CRS 
^ question numbers) . ^ ' 

Services: 

46. Mental Retardation ^ 

47. Vocational and Other Training 

( - 

48. Maintenance V 

49. Transportation . i 

50. Services to the Family 

51. Specialized Services for Blind, Deaf, Severe Disabilities 
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Tabic 44 (cont.) 




• 




• 


1 

* * 




DOliS Tllli CASli RECOIL) DOCUMENT. THAT 
Till- Sr.RVICK WAS (IIVEN. . . • 


UE 


MI 


MS 


'or . 


. PA^ 


VA " 


^7 - Tr* 1 prniniTiiitifi m - 

tions^ • 


Performance (%) 














53 - Occupational 
licenses^ tools, 
equipment 


Performance f%) 










• 




54 - Other goods 
and services? 


Performance f%) 


33 




60 










N = 


120 


119 


119 


100 




* 

100 














f 




* Virginia and Pennsylvania were able to 


compute item 54: Pennsylvania - 


- Performance - JL^OO 
Consistency Ratio - 


.917 








Virginia -- Performance - 100 

Consistency Ratio - -.000 
/ . 




^ 








322 




( 




ERLC 


1 








4 







303 



CRS QUESTION WORDINGS 



/ 



The exact wording of the Case Review items for Section I IB, per- 
taining to Standard 12,. appears below. 



Item C: Does the case record document that the service was given to 
the ;clfent?' ^ , 



Item C is asked in^ reference to the following services (keyed to CRS 
question numbers). 

# 

Services : 

'52 . Telecommunications ^ 

53. Occupational Licenses, Tools, Equipment 

54. Other Goods and Services 
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Section V: Individualized Written Rehabilitation Program Status 12 ^ 

In a sense, the purpose of the HVRP is to establish an alliance between* 
the agency and the client for the provision of certain services toward the 
achievement of a specific vocational goal. As such, it is important that 
the Il^RP contains all the information necessary to establish such an 
alliance. Section V attempts to document, among other things: 

• that the client was informed of the terms and conditions for the 
provision o£ services; 

thlt the client was informed of client Tights; ♦ 

' • that the cl:^nt participated in the full planning and review process; 
and 



al>i^n 



• that the IWRP contains essential informabi^n such as goals, time- 
frames, evaluation procedures and schedules, etc. 

Provision of this information clarifies the roles, relationships, and 
duties of agency and client toward achieving the vocational goal, the essence 
of the IWRP process. 

Analysis of the pretest data for Section > V (see Table 45) indicates that 
for the most part the IWRP's contained the required information. Items 1-17 

documenting that the IWRP sets forth conditions for the provision of 
services, informs client of client rights, and contains necessary material 
sucjj as goals, services to be provided, and estimated timeframes generally 
show high performance le\^els and consistency ratios, with a few exceptions. 
Among these exceptions are the items concerned with the IWRP's descriptions 
of the extent of client participation in the cost of services Cit:em 36); and 
with documenting that the client was informed of the right to be fully 
consulted regarding change's in the program (item 4a) . ^ Performance leve^ls 
and consistency ratio? for these items show that current documentation 
procedures do not always give an unambiguous indication that clients were 
informed of cei^tain conditions and rights. * 

Items 18-26 show a ^marked reduction in both performance levels and ..^ 
consistency ratios,^ compared to items 1-17. Items 18-26 pertain tijaocumenting 
evaluation criteria and procedures, IWRP reviews, termination procedures, 
and post-closure services. Considering that .these items seek to ensure 
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Table 45 

Section V: 1 nil i vidua Tiz-cd Written Rchabi 1 i tat/on i>rogrum - Status 12 aiiJ Above 



DOr-S Tllli CASli lUXiOiU). . . 


'/ 


MI 


MS ' 


OR 


• PA 


VA 


1 - Contain an IKRP? 


i'erformance (%) / 


( 97 


98 


94 


100 


99 


99 


Consistericy Rati/ 


1 .000 


.970 


.986 ■ 


1 

.967 


.917 


• .931 . 


IJOliS Tllf: IWI<i\ . . • / 










2 - Present basis 
tor determination 
of eligibility? 


Performance Pi) 


99 


100 


98 


100 


100 ^ 


100 


Consistencw Ratio 


, 1.000 


.924 


.542 


.967 


.933 


.833 ' 


3a - Show client 
responsibilities for 
provision of sjprvice? 


Performance (%) 


94 


97 


96 


98 


100 


100 


Consistency Ratio 


.963 


.970 


, ^625 


.983 


.983 


.833 


3b - Show extent of 
client participation 
in cost of services? 


Perfo/mance {%) 


90 


85 ' 


55 


94 


99 


55 


Con/istency Ratio 


\.907 


.818 


.556 


.983 


.967 . 


.792 , 


4a - Document client 
was informed of right 
to be consulted over 
: IWRP changes? 


Performance (%) 


97 


^ 

12 


98 


59 


100 


100 


Consistency Ratio 


.981 


.848 


.625 


.683 


1.000 


■'.800 


4b - Document client ; 
was informed of righw 
to review if dis- / 
satisfied? / 


Performance (%) 


97 


92 


,98 


94 ^ 


^00 


ibo 


Consistency Ratio 


1.000 


.848 


.625 


.933 


1.000 


.800 



\ 
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CRS QUESTION WORDINGS 

The exact wording oi the Case Review items for Section V, per- 
' tairiing to' Standard 12, appears below, 

» 

r 

!• Is there, an IWRP in the case record? 



Does the IWRP (state form).,. ■ " , ^ 

12. present* tlie general basis for a determination ^of eligibility? 

y 3e set forth the te^Tiis and conditions for the provision of . 
•services, including: * 

a. client responsibilities in carrying out the 
" program, such as cooperation, attendance, etc.? 

- b. the extent of client participation, in the cost 
of services? . 

4, document .that the client was informed' of client rights and 
' remedies, including: . • 

a. right to be fully consulted regarding any 
changes or amendments in, the rehabilitation 
.program? ^ 

b. the right to administrative review i-n case of 
dissatisfaction with services? r / 
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'Table 45 (cont.) 



DOES TilE IWRP, 


DC 


MI 


MS X 


OR 


. .PA 


VA 


5 - Show^ that IWRP , 
v^as separate part of 
case record? 


. Performance (%) 


100 


99 


99 


100 


1 

98 


67 


Consistency Ratio 


.981 


.970 


^~T~T 

..528 


«.983 


7* 

.933 


.800 


6 ' J^i^ovi client re- . 
ceifed^opy of IWRP 
and amendments? 


Performance (%). 


97 






91 


89 


> 

99 


Consistency Ratio 


1.000 


.970 


— ^ 

.847 \ 


.983 


.917 


— < 

.944 


9 - rndica|:e program 
was developed with 
consultation? 


Performance • (%) 


90 


^ 24 


89 • 


\\' 92 


98 


97 


Consistency Ratio 


.907 


• " .758 


.958 


' .-900 ■ 


■ .900 


.944 


11 - Place emphasis^ 
on vocational goals? 


J/evfoTmance (%) 


9j6 . 


96 




'^'•99 ■' 


94 


98 


Consistency Ratio 


.926 


.970 






- ..933 


* .958 - 


12 - State long-term 
employment goal? 

: t 


Performance (%) 


97 


\ 98 


95 




97 '. 


100 


Consistency Ratio 


.981 


.970 


.986 


'.950 


.950 


.986 


13 -*^tate interme- 


Performance (%) 


" ' 91 


17 


79 


. 90 ^' ' ■ 


^■93 


■ 95 


diate rehabilitation 
objectives? , 


Consistency Ratio * 


.907 


.879 . 


.819 




.833 


■V ■■ 
.983 


.903 



o 

00' 
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CRS QUESTION WORDINGS 



The exact wording of the Case Review items for Section V, per 
taining to Standard 12, appears below. 

Does, the IWRP (state form).;. " 

. 5* reflect that the IWRP was maintained as a separate part of 
•the cas^'e recqrd? 

♦ 

*6. show that the- client received a Cdpy of the IWRP and' 
substantial amehdments? 

9* indicate that thejprogram was developed and amended with 
the client's participation or, as appropriate with the 
parent, guardian, or other representative? 

11. place primary emphasis on the determination and achieve^- 
ment of a vocational goal? 

12. state the long-range ertiployment goal? 

13. state the intermediate rehabilitation objectives? 



7 
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Table 45 (cont.) 



DOES THE IWRP... 


• DE 


MI 


MS 


' OR 


PA 


VA 


14 - State services 


Performance (%) 


95 


86 


91 . 


92 


95 


96 


necessary to achieve 
goals? 


Consistency Ratio 


..926 


• 758 


.889 


.950 


.917 


.944 


15 - Contain begin- 
ning date for each 
. service? 


Performance (%) 


97 


72 . 


84 


89 


87 


98 


Consistency Ratio 


• 926 


• 788 ^ 


.931 


.883 


.733 ■ 


.958 


16 - Contain anti- 
cipated duration of 
each service? 


Performance (%) 


9/ 


58 


76.. 


. 88 


85 


92 


Consistency Ratio 


• 907 


•1^ 


'.931 


.850 


<t 

• .717 


^ .875 


17 - Provide time 


Performance (%) 


91 


85 


' 54 


97. 


89 ' 


89 ' " 


frame goals may^ be 
achieved in? ^ , 


Consistency Ratio 


• 796 


^ ^803 


.681 


— .917 


.867 


.847 




Performance (%) 


80 


36 


25 


80 


63 


' 88 


18 - State evaluation 
criteria? 


Consistency Ratio 


.778 


• 803 


.708 


.700 


.583 


.875 




Performance (%) 


83 


74 


19 


81 


/ s56 , 


95 


19 - State evaluation 
procedure? j 


Consistency Ratio 


• 926 


• 697 


■t 

.722 


'■ — ■ f 

.767 / 


♦ .*67 


.944 












> 







. 332 



311 



CRS QUESTION WORDINGS 



The exact wording of the Case Review items for Section V> per- 
taining to Standard 12, appears below. 

Does the IW^P (state f ormO . . . 

14/ state the specific VR services to be provided' to 

achieve the intermediate objectives^ and the employment 
goal? 

15. contain the projected date for the initiation of each '* 
service? 

16. ' contain the anticipated duration for each service 

planned? 

17. provide the projected time within which rehabilitation 
objectives and goals may be achievec}? 

18. state the objective criteria upon which an evaluation of 
J the client's progress toward an employability goal is 

based? 

19. state the procedure by which the client is evaluated? 



\ 
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Table 45 (cont.) 



DOnS Tflli IKR1\ 


DO , 


MI 


MS 


'■ OR 


•L. PA 


VA 


20 - Give schedule 
for reviews and 
evaluation? > 


Pi3rformance {%) 


74 


71 


31 


' "so 




9.7 


Consistency Ratio 


.852 


.742 


.681 


7-17 - 


.4i^600 


.917 


21 - Contain results 
of reviews and evalu- 
ations? 


• 

Performance (%) 


28 


45 


'17- 


73 


47 


17 


Consistency Ratio 


.704 • 


.576 


.750 


.683 


to. 500 


.90S 


22 - Contain formal 
annual review? 


Performance (%) 


9 ^ 


43 


21 


36 


73 


ftO 


Consistency Ratio 


.759 


.773 


.650 


. .750 


.750 


.764 


23 - Contain client 
views? 


Performance (%) 


78 


27 


79 


84 


95. 


90 


Consistency Ratio 


ft 


•7 10 


. 847 


■ • .733 


.917 . 


.847 


DOliS.THE CASE RECORD. . 


, ♦ 














24a - Show termina- 
tion was made with 


Performance (%) , 


83 


74; 


100 


71 


# ^ 

67 


"■ • 74 . 


consultation with 
client? 


Consistency Ratio 


.630 h 


.424 


.694 


.j694 


.750- 


.583 . 


24b - Show termina- 


Performance (%) - ' > 


100 


93 


82 


"lOO 


98 


98 


tion rationale was 
amendment to IWRP? 


'Consistency Ratio 


.^63 


• .515 


.667 


:917 


".875 


.889 
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CRS QUESTION WORDINGS 



V 



The exact wording of the Case Review items for Section V, per- 
taining to Standard 12, appears below. 

"poes the IWRP (state form)... 

20. contain a schedule for the periodic reviews and progress 
evaluations? * ' ' Ji 

21. contain a record of the results of the scheduled reviews 
and evaluations? 

22. show that a formal, annual review has been conducted, if 
the IWRP has achieved at least first anniversary status? 

23. document the client's views, or as appropriate, the views 
of the parent, guardian, or other representative concerning 
the goals, objectives, and VR services, being provided?. 



Does the case record... 



/ 



24a. show that the decision to terminate services was made in full 
consultation with the client or as appropriate, with the 
parent, guardian, or other representative? 

24b.' show that the rationale for the decision to terminate services 
was recorded as a certified amendment to the IWRP? 



o 



Table 45 (cont.) 


i 










■ 


'\ 




DOES THE CASE RECORD... 


DE 


MI 


MS 


OR 


PA 


VA 




24c - Give certifica- 
tion of . ineligibility' 


Performance (%) 


25 


48 


0 


ioo\ 


' 31 • 


95 ■ 




Consistency Ratio 


.630 


.394 


.694 - 


.917\ 


.667 


.889 




24(1 - Show pi^ovision 


Performance (%) 


' 24 


6 


18 


n V 


35 


98 




was made for periodic 
review? 


Consistency Ratio 


.741 


.636 , 


.611" 


.556 








DOES THE CLOSURE STATEMENT... 


- 














25a - Give descrip- 
tion .of basis for 
rehabilitation? 


Performance 


92 


98 


87 


98 


95 


■■■98 


M 


Consistency Ratio 


..'926 


■ .909 


, .771 - 


1.000 


.806 


.958 ■ 














\ 


26'- Is there a Post-^ 
Employment amendment 
-.to IIVRP? 


Performance 


, 0 


2 


0 


0 


0 


■ 6 




Consis'tency Ratio 


.644 " 


.78.8 


..750 


.861 


.783 


.708 




N = 


. 120 


119 


■119 ■ 


■ 100 


98 


118 




) 

7 




\ 

\ 
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CRS QUESTION. WORDINGS 



* The exact wording of the Case Review items for Section V, per- 
taining to Standard 12, - appears below. * , 

Does the case record.. • . * * . - ' 

• 24c, show that a 'certification of ineligibility was then 
executed? 

24cl. show that the provision was made^ for a periodic review, 
at least annually, of/the ineligibility decision? 



Does the closure statement../. 

25a. ^ description oi the bas 
determined to ^e 




on which the client was 



7 * 
26/ fs ther^' an amended IWRP for Post ^Employment Services? 



/ 
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that, client rights are being protected, dnd are important; to an overall 



<,-— — ■ — - ~ — — » — — — — 

evaluation of the agency, fully documenting these Items is very important* 

Section VI': Delivery of . Services Statuses 14, 16, 18; 20, 22, and 32 

This section complements the information provided^ in Section V* When 
these two are taken together, they describe the overall VR process, con- 
sisting of the plan {i.e^^^^^^^e terms, conditions, and information set forth 
in, the IWRP needed to provide", services), and the specific program of seinrices 
undertaken to achieve the vocational goal embodied in the IIVRP. Tables 46 ^ 
and 47 show the frequency with which particular services were planned and 
'delivered. Our interest in thesQ tables is not with performance levels 
per se, since there is no .basis to judge performance as "good" or 't)ad". 
Rather, we focus here on the consistency ratios, to give an indication of 
the clarity of ducmentation. Concentrating on consistency* ratios, there are 
two services that show abnormally low readings. There are evaluation and 
diagnostic services (item 1 on both tables) and counselling and guidance 
services (item 2 on both tables). For evaluation and diagnostic services, the 
consistency ratios are well below the required .750, with one exception. 
Tliis is interesting, since most applicants receive some sort of evaluation^ 
and diagnostic services to determine eligibility. Counselling and guidance 
services also showed low consistency ratios, but to a lesser extent .than 
item 1. This is even more surprising, given the^ key role of counseling 
in VR. 'Counselors must understand that clear documentation is needed even 
for "core" services delivered as a matter of course. 

Section VII: Termination of Case 

The questions in Section VII, relevant tb Standard 12, seek to ensure: 

^0 that the rationale for closure decisions are recorded on^ the IWRP; 
and 

• thavthe client (or his/her appropriate representa'tive) was 
consulted prior to the closure decision. 

Provision of this information ensures that the agency has a source from 
which it can draw information about past ineligibility decisions* to make sure 
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they were made in a consistent manner and that client rights were protected 
during the closure process. ^ * . ^ 

Table 48 presents the results o£ Section VII for status 28 and 30 
closures. (Again, as with Standard 10, no d^ta was available on 08 closures.) 
For item 34, most states provided the 'rationale fbr ineligibili^ as an 
amendment to the program in most cases. ' The lower levels of performance ^nust 
be interpreted wjj^th caution, given that their consistency ratios are well 
below .750» ^ •'^ ' . 

Itejn 35, however, presents more of ^ problem, with only one consistency 
ratio above the .750 required for a valid result. This would **seem .to indicate 
that documenting consultations wifh .clients is difficjilt as the system 13 
currently set up. Considering that protection 'of client rights must be a 
paramount concern of a state VR agency, a more useful documentation procedure 
may be in order. - ^ . • ' 

■■ ; " . ■ 

I 





Table 46 : 

Section VI: Delivery of Services - Statuses 14; 16, 18, 20, 22, and 32 . 
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DOI-S THE CASH RfiCORD D 
SJiRVICfi WAS I>l,AN,Ni:i) I-O 


OClJMIiNT THAT THE 
R TIIH CMENT? 


Dr; 


MI 


MS 


■ OR 


PA 


VA 


1 - livaluat ion ;and 
diagnostic services 


Performance (%) 


51 


45 


52 


41 


78 


44 


Consistency Ratio 


.685 


.697 


.597 


.600 


.650 • 


.681 


2 - Counseling and 
guidance 


Performanc/e (%) 


61 


68 


61 


23 


92 


98 


Consistency Ratio 


.926 


.682 


.583 


.683 


.967 


.972 


3 -Physical restora- 

LXVJll 


Performance (%) 


43 


, 21 


81 


21 . 


60 


34 


Consistency Ratio 


.94.4 


.955 


.944 ■ 


.900 


.833 


1.000 


4 - Mental restora- 
tion 


Performance (%) 


24 > 


16 


■0 


1 


21 


16 


Consistency Ratio 


.944 


.803 


.903 


.933 


.783 


.972 


5 - Vocational and 
other training 


Perrormance {-ej 


63 


57 


27 


79 


33 


S9 


Consistency Ratio 


.944 


.8^)3 


.667, 


.917 


.750 


.958 




Performance (%) 


' '2F 


25 


12 


19 




21 . 


6 - Maintenance 


Consistency Ratio 


.963 


.939 


.931 


.933 


— % 

.800 


.986 


7 - Transportation 


Performance (%) 


' 43 


39 


20 


56 


V 

4 


25 


Consistency Ratio 


1.000 


.894 


.819 


.783 


■ .817 


.986 
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CRS QUESTION WORDINGS 



The exact wording of the Case" Review items for Section VI, per 
taining to Standard 12, appears below. 



Item B: Does the case record document that the servic^ was planned 
for the client? 



Item B is asked in reference to the following services (keyed to CRS 
question numbers) . 

Services : 

1. Evaluation and Diagnostic Services 

2. Counseling and Guidance 

3. Physical Restoration 
•4. Mental Restoration 

5. Vocational and Other Traii^dng 

6. Maintenance ^ 

7. Transportation 
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DOES THE CASE RECORD DOCUMENT THAT THE 
SERVICE WAS PLANNED FOR THE CLIENT? 


DE 


MI 


MS 


OR 


PA 


VA ■ ■" 


8 - Services to the 
family 


Performance (%) 


2 


1 


1 


• 

0 


1 


■ 0 


Consistency Ratio 


.981 


1.000 


.917 


.983 


.800 


.986 


9 - Specialized 
services^ 


Performance (%) 


4 


3 


11 


2 


3 


2 


Consistency Ratio 


i.ood 


..970 


.819 


1.000 ' 


,783 


• .972 


10 - Tele-communica- 
tion 


Performance 


2 


l' 


3 


0 


1 


-1 


Consistency Ratio' 


1.000 


.985 


.903 


1.000 


.800 


.972 ■ 


11 - Occupational 
equipment 


Performance (%) 


4 


6 


7 


16 


■ 1 


5 • 


Consistency Ratio 


1.000 


.909 


.917 


1.000 


.800 


.931 


.12 - Other goods 
and services 


Performance (%) ' 


8 


• 12 


13 


18 


4 


3 


Consistency Ratio 


.926 


.833 


.875 


.•900 


.800 


.903 


13 - Placement 


Performance (%) 


50 


40 


23 


51 


65 


93 


Consistency Ratio 


.778 ■ 


.773 


.806 


.76.7 


.600 


1.000 




• 

Performance (%) • 


1 


Missing 


Missing 


Missing 


Missing 


Missing 


14 - Post-emplojnnent 


Consistency Ratio 


.926 


Missing 


Missing * 


Missing 


Missing 


Missing 


N = 


120 . 


119 


119 


•100 - 


98 


118 
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CRS QUESTION WORDINGS 



The exact wording of the Case Review items for Section VI, per- 
taining to Standard *1 2, appears below^. _ 



Item B: .Does the case record document, that the service was planned 
for the client? 



Item B is asked in reference to the following services (keyed to CRS 
'question numbers). • . . ^ 

Services: 

8. Services to the Family 

9. Specialized Serv3.ces for Blind, Deaf, Severe Disabilities 

10. Telecommunications 

11. Occupational Licenses, Tools, Equipment 

12. . Other Goods and .Services 
15. Placement 
14 • Post-Employment 



r 
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Table' 47 

Section VI: Delivery of Services - Statuses- 14, 16, 18, 20, 22, and 32* 
^ t 


SliRVICli WASiClVliN TO TlHi CLIIINT? . ' 


DI- 


« 

MI 


MS 


OR . 


PA 


VA 


1^- Evaluation and 
diagnostic services 


Performance (%) 


SS 


47 


81 


41 


75 


45 


Consistency Ratio 


.833 


.652 


.556 


.617 


.633' 


".708 


2 - Counseling and 
guidance 


Performance (%) 


61 


73 


79 • 


19 


85 


■ 92 


Consistency Ratio 


\889 


' .727 


.611 


.733 


.817 


.958 


3 - Physical restora- 
tion 


Performance [%) 


41 


19 • 


92 


21 


■ ' 64 


. 34 


Cpnsistency Ratio 


*. .870 


' .939 


.958 


.917 


' .733 


• .986 


4 - Mental restora- 
tion 


Performance (%) 


. 20 


18 


6 


' . 0 * 


•27 


13 


Consistency Ratio 


.907 


.833 


.875 > 


.917 


.617 


;-972 


5 - Vocational and. 
other training 


Performance (%) 


63 


63 


57 


71 


36 


55 


Consistency Ratio 


.926 


.818 


.639 


.800 


.567 - 


.889 ■ 


6 - Maintenance 


Performance (%) 


.25' ' 


24 


43 


19 


8 


18 - 


Consistency Ratio 


.944 


.924 


.875 


.917 


.567 


.958 


< 


• Performance {%) 


40 


35 


55 


52 


4 


24 


7 - Tranportation 


Consistency Ratio . 


* .981 


.818 


.764 


.733 


.550 


.972 



.'(^RS QUESTION WORDINGS ,\ ' ' 

The exact wording -of the Case Review items for SeptioaVI, per- 
taining Standar4 12, appears below. ^ , 



Item C: Does the c^se record document that the servicfe was given 
tp the client? 



V 



Item is' as|ced in reference 
question numbers) . • 



to the -following services (keyed to CRS 



S'ervices : ' 

1. Evaluation an^ Diagnostic Services 

2. ^ Counseling and Guidance. » - 

3. Physical Restoration 
^^^4. Mental Restoration 

5. Vocational and Other Training 

5. Maintenance 

7, Transportation 
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DOES TllO CASK RECORD TOCUMENT THAT TTIE 
SERVICES WAS GIVEN TO THE CLIENT? 


DE 


MI ' 


MS 


OR 


PA ^ 


VA 


8 - Services to the 
family 


Performance (%) 


1 


■ 0 • 


14 


0 


0; 


0 


Consistency Ratio 


■ .981 


1.000, 


.861 


.983 


• -.567 


• .972 


9 - Specialized 
services 


Performance (%) 


. 4 . 


3 


35 


2 


2 


2 


Consistency Ratio 


1.000' 


.985 


.778 


.983 


.550 ' 


.972 


10 - Tele-communica- 
tion 


Performance* (%) 


2 


1 


11 


0 - 


0 


1 


Consistency Ratio 


1.000 


• .985 


.861 


. .983y' 


.567 


.958 


.11- Occupational 
equipment 


Performance (%) 


4 


7 


29 


14 


0 


5 


Consistency Ratio • 


1.000 


!909 


.8-75 


.1.000 


.567 


.•958 


12 - Other goods 
and services 


Performance (%) 


8 


9 


47 


19 ■ / 


6 


3 


Consistency Ratio 


.944 


-.833 


.861 


.900 


■ .567' 


.889 


13 - Placement 


Performance (%) 


47 • 


72 


49 


21 


38 


60 


'Consistency Ratio 


.759 

— >■ 


.788 


.750 . 


.700 


.517 


.694 




Performance (%) •. 


0 


Wissing 


13 


■ 1 


0 


1 


14 -.Post-employment 


Consistency Ratio 


■833^ 


Missing 


.875 


.983 


.567 • 


.944 




;r'^2o. ; 


119 


119 


100 


98 


118 
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. _ CRS QUESTION WORDINGS 

Hie exact wording of the Case Review items for Section VI, per- 

tainihg to Standard 12, appears belo;/. 

V ' ' ' , ■ 

> ' 

Item C: Does the case record document that the service was given 
to the client? ^ ' ^ 

Item C is aske^ in reference to the following services (keyed to CRS 
question numbers) . 

Services: 

8. Services to the Family 

9. Specialized Services for Blind, Deaf, Severe Disabilities 
ip . Telecommunications 

11. Occupational Licenses, Tools, Equipment 

12. Other Goods and Services 
13'. Placement 

14 . Post-Employment - . . 
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Table 48 * - 

Section VI J: lermination of Cases - Statuses 30 and 28 



DOl-S Tllli CA*S1-: RKCORD... 


Dli 


J 

MI ^ 


f * 

MS 


• OR 


P-A 


VA 


34- - Xontaiii. ration- 
ale for, eligibi lity ' 
-.or amendmeat to 
program? 


Performance (%) 


92 


70 


71 


100 


83 


98 


Consistency Ratio 


.8"! 5 


.455 


.583 ■ 


.972 


■ .833 


.917 


35 - Show ineligibil- 
ity was made after 
client consultation? 


Performanqe (%) 


78 


59 


81 


70 


77 


74" 


Consistency Ratio 


.630 


. .485 


.722 ■• 


.722 


.817 


.667 


N = 

• 


120 


119 


119 


100 


98 


118 
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■ , CRS QUESTION WORDINGS 

. The exact wording of the Case Review items for Section VII, per- 
taining to Standard 12, appears below. 

Does the case record... 

34.* contain the rationale for the ineligibility determination 
- as an amendment to the program? 

35 • show that the ineligibility determination was made only 

after full consultation with the client, or as appropriate, 
with the parent, guardian, or other representative? 



^This is the same wording as used in. item 20 (not shown) pertaining to 
clients closed 08 from 06; except that the word ''IWRP'^'is substituted 
for ^'program/* 
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R->500 VERIFICATION 

The R-500 Verification procedure is designed to respond to Standard 9. 
It assesses the degree to which information submitted to RSA on critical 
items of the R-300 was corroborated by casef i Ie~ Tnformation . ~ ~ ' 

All of the data items on the R-300 Verification instrument (a copy of 
which follows as Exhibit iO) are necessary to ensure the integrity of the 
data source on which many of the Performance Standards' data elements are 
based. On(5 of the purposes of Standard 9 is to verify the R-300 information 
such that users of the R-300 data can^ have confidence in the accuracy of 
the data reported to the states, the department admini3tration, and the 
Congress. 

In this pretest, "verification of accuracy" refers to a manual confir- 
mation procedure intended to insure that file information supports and 
corroborates the R-300 documents. To minimize the burden of this procedure, 
only those R-300 items proposed as data items in the Performance Standards' 
data elements were "verified" by this process. 

Training and Logistics 

The steps in the R-300 Verification procedure, which in operation 
usually also involve assessing timeliness of case movement, were as follows: 

# Pre-training Logi'stics 

Negotiate dates with states (usually coincided with SDSU's 

♦ 

follow-up^ visit) 

lit i lire same sample selection and dimensions as for CRS 
Review te^am se-lect'ion (4-10 reviewers): 

* Delaware - 3 administrators, 1 program evaluator 

(R-509 and Timeliness) 

- Michigan - 10 office managers (R-300) 

4 program evaluation unit staff (Timeliness) 

i 

* Mississippi - 2 administrators, 5 program evaluators 

' (R-300 and Timeliness) 

.* Oregon - \i2 CRS reviewers (Timeliness) 
f^; 4 program evaluators (R-300) 

* Pennsylva)iia - 12 CRS reviewefs (Timeliness) , / . 

10 case review specialists and program 
- .evaluators (R-300) 

* Virginia.|- 3 , administrators , 3 program evaluators, 

' V 1 secretary (R-300 and Timeliness) * 
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TIME STARTED 



^ 2 3 4 5 6 7 8 9 10 U 12^ 



R-300 VERIFICATIONS 



Prepared by 

Berkeley Planning Associates 

1912 Bonita Avenue 
Berkeley, California 947.04 
y (415) 549-3492 • 



TO Be Used, 
in the 6 Model Units 

ONLY ^ 
for Purposes of ^ 
PRETESTING THE PROCEDURAL STANDARDS 

February. 15, 1980 
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EXHIBIT 10 (continued) 
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SECTION I: CASE INFQR^tATION 
A. Caso Identification 



ANSWER COLITIN 



1. Reviewer Nunber 



2. ReviQucr: 

(1) State 

(2) Federal 

(3) Other (specify)_ 

^ P»vT^w»y 



(1) Adsifnistrator 

*(2) Supervisor 

(3) Counselor 

(4) Clerical 

(5) Otiier Specify) 

Date of Review^ 



5. State Agency (code): 

6. Case .Nonber (right justify); 

7. Closure Status: 

CS 



(n 
C) 

(5) 

(5) 
C6) 



C3 

08 
26 

:3 

30 



ron 00 
rcn 02 
ron 06 



Does the client*s case record contain a copy aft h^ case 
service report R-300 or the equivalent state statistical 
reporting fortn(s)? 

(1) ves 

(2) no 



InOTE: important INSTRUCTIONS " . 

Respond to items 9-32 by filling in the number in the 
I answer column that Indicates your findings relative 
to the agreement of dat-a found on the' R-300 '(or its 
ecu IV a lent) to that found in the case record '(or 
other data source, when specifically indicated'). The 
1 fopov»ir\tl is. a delineation of potential responses arid 
I their detinitions: 



,1 1 -t ■ 
Case record contains iijfor- 
mation consistent with that 
found on the J^-300 (or 
equivalent) . 

,R- 300' Contradicted Case record contains infor- 
'm^rlidi^different from that 
/ound on the R-300 (or 
/ equivalent) . ^ 

Data (including zeros) re- 
corded on R-300 (or equi- 
valent onl y ; no informa- 
tion in case record, 



1 = P.- 500 Contimed 



5 *^ R-300 ,Only 



4 a R-300 Missing 



N*oQlti^'*(bIanks)' on R-300 
(or equivalent). ^ 
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, MO. . Day Year 
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SCCTIO.N {: C\SV. {NIOliMATION 
R-300 Vcrit'icj ci o n 



Cont i lutccl 
P'i''c 2 



u%.n .ka. jouKd tne case record (or other data source, when indicated)? 

■ \ 

9. The client* s age ^ 

10. The client's Social Security number ^ 

11. The client's SSDI status at referral 

■ " ■ f 

12. The client's SSI status a,t referral 

13. The client's major (primary)' disabl ing condition 

^^^^ disabling condition (use the disability code 
book for verification) ^- ^ 



1: 



The client' s, secondary disability 



16. The disability code for that disability (use the disabi 1 ity -code book 
' for verification) \ , 

17. Previous closure of the client within 36 n^ths , 



IS. If previously' do s,ed, the outcome (rehabilitated oV npt> rehabilitated) 
of the client (if not applicable, fill in 4) \ ' 

19. The clioht's work status at referra.1 



20. The client's weekly earnings at referrarl 



21., The 



^^^\^^'s primary source of support .at referral 



\ 1 



22. Identification of client as TF (Social Security Trust-FundJ on the 
Feder.il .Special Progran Identification at closure 

(Ll>e official transmittal from SSA for verification; 
if blank/ on R-30p and no transmittal, fill in 1; 
if bLu>j( t>n R-500 and transmittal, fill in 2; 
if choci? on R-300 and transmittal, fill in 1;'' 
if check on R-300 and no transmittal, fill in 30 

^23: I;J^"^it'ication of client as SF (Supplemental Security Income Fund] on 
the Federal Special Program Identification at closure 
(Use official transmittal .from ^SA for verif icatio»i 
if blanj. on J^-3^and no transmittal, fill in 1; 
if blank on R-3fr" ind transmittal." fill in 2;, * 
if check on R-300 and transmittal, fill in 1^ " . ' 

if check on R-^Z^Q and no transmittal, fill in 3 ) 

21. IJcnt'ification.of client as SD (Sever.ely Disabled) on tHe Federal Special' 
f roi^tjm IJ'entvtication at closure 

(M^c criteria' ^ta^llal for' verif icationr 

if blank on R-30,d and doesrt't meet criteria = 1; 

if blank on R-306 and meets criteria = 2; *• 
*it"^c]!jQck on ft. 300 itnd meets criteria = 1; ■ 
It' check on R.300 and doesn't meet criteria a 2.) 

25. . The client's, SSDi status at closure • 
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Exhibit 10(cont.) 

— r ■ 

SECTION I: CASE INFORMATION 
R-500 Verification 



Continued, 
Pa-ge 3 



J3 the follouing inforvntion on the R^ZOO, or its equivalent^ consistent 
vith that found in the case record Cor other data source, when indicated)? 

26. The client's SSI status at closure 

27. The client's work status at closure ♦ 

J^- ^ 

28. The client's weekly earnings' at closure 

29. The number of months from client acceptance 'to^ closure (Status 10-24) 

30. The client's outcome status * 

31. If closed 03, :8 or 30, reason for"^ nonrehabilitated closure (if closed 
26, fill i;i 4) ' . ' 

i " * 

32. The total cost of all case services (within a W-i margin of error) - 



0 -a 

c c 
o o 



C 

•H 

•-4 M 

8 S 



o 
o 

1 


o 
o 

t 

06 


o 
o 
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i 
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o 
o 
n 
1 

Pi 
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0 


Q 
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® 




Q 


Q 




® 


G) 


@ 


(D 


0 


Q 


@ 


® 


0 


® 


Q 




0 


G) 


Q 




0 



To figure 10% Margin of Error 

a) Total cost of case services, 
according to case record. . . . 



b) Multiply (a) by 10% ..... . 

c) Add (b) to (a). . 

d) Subtract (b) from (a) * 

e) Total cost of case services, ' 
according to R-300 $^ 

f) If (e) falls between (c) and 
(d), the R-300 is "confimed" 
si. If (e) is greater than . 
(c) or less than (d) , the R-300 
is "contradicted" = 2. 



33. CC^WCNTS: 



3t. Len(:tli of tine in minutes necessary to complete Section I: 
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On-site Training 
Two' to three hours of training in R-300 Verification and 
Tl]niBine-ss--Ais>ss^ based on a process of identifying 

the" appropriate R-300 input documents in the state's case 
file and establishing what constitutes 'if orroborating infor- 
ination" on an item-by-i-item basis. \ 

Individual application of R-300 Verification and Timeliness ' 
Assessment instruments to representative training case. 
^ Large group discussion to re\olve discrepancies and develop 
consistent judgments. ^ V 

• R-500 Verification and Timeliness Assessment 
Standardized method of distribution of cases. 
Integrated distribution of same six reliability cases to ~ 
all reviewers to determine inter-rater consistency. 
Case flow monitored throughout actual cas^ reviews. 

Issues Raised During the Pretest 

The following discussion presents a synthesis of the comments made by 
the MEUs as they pertain to various aspects of yt1ir"R^0 Verification pro- 
cedure. These* include reviewing problems wit/ the instrument and/or the 
procedure', the perceived utility of the information generated, and the 
particular concerns pf the agencies regarding the feasibility of implement- 
ing this procedure using current data- collection systems. 

/ ' ' ' 

Cost and Effort 

/ 

Reg^«^HVg -the-^ost and effort involved in verifying the R-300, concern 

was expressed over the increased time and paperwork requirements placed on 
administrative and clerical staff. Also, the logistics of having case files 
removed from the district office for extended periods of time was seen as 
problematic, especially if these case files were needed in the interim. But 
the? cost and effort involved in the R-500 Verification was seen as a lesser 
burden than that imposed by the Case Review Schedule. For this reason, s<^6 
agcncies^saw no problems in conducting the R-300 Verification on an'annual 
basis. 
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Comparison to State Methods 

The concerns of cost and effort are directly related to the agencies'* 
perception of the adequacy of their data collection system as cbmpared to 
the requirements of the R-300 Verification procedure. In many states, the ^ 
data collection system as it is currently maintained does, not contain the 
type, of information defined as corroborating an R-SOO item. 

Table 49 illustrates the "unverifiable items'* for each.MEU, In some . 
instances (Pennsylvania primarily) the R-300 equivalent was the input form 
for the R-300 data, and efforts had been made to discpurage duplicating 
information in the files which was already there. In other instances, .spme 
information functions had been totally transferred to other departments in 
the agency (e.g., cost information, Trust Fund ^d Supplemental Security 
Income Fund information), and are absent from the ease files. In four states, 
the case file information did not permit a determiSi^tion^^ whether the client 
had been previously closed by the agency and,. if so, in wha\- status. For 
three states, the computer calculates tjie client 's^ months in status 10^24, 
and the information is missing from the input document. In Michigan and • 
Oregon, Trust Fund and Supplemental Security Income Fund information is 
handled elsewhere in the agency, as is cost information in Oregon and Virginia, 
No procedures were developed for the pretest to deal with other branches of 
state bureaucracies controlling this information. Rather, consistent codes . 
were assigned in these instances. 

Other comments noted that the verification procedure is designed for 
RSA, not the states. This is a correct observation; R-300 Verification is 
designed to ensure that the information submitted to RSA is reliable and 
valid. For thi^, reason, we do not advocate that states be allowed to delete, 
particular R-300 items from their verification procedure, but of course they 
could add items for verification if they consider them to be of greater 
importance to the agency. 

Possible Benefits . * , , ' 



There were many possible benefits which reviewers believed might result 
from ongoing implementation of the R-300 Verification procedure. The most 
frequently mentioned benefits were the following:^ 



ERIC 



. . '336 

* . Table 49^/^; . • 
R-300 Verification: Unverif ial)le Items ' 



R-300 
Item 


Verification 
Item Number 


DE 


MI 


MS 


OR. , 


PA 


VA ': 


I.E. 


9 


I 






' X 






■ 2. A. 


10 












~ - 


2.B. 


11 












X 


2.C. 


12) 












>x 


2.1.1. 


13 














2.1.1. 


14' 






X 








2.1.2. 1 "is 










• 




2. J. 


17 

* 


X 




f. . 


X 




■ X " 


. 2. J. 


18 


X 






X 


X 




2.0. 


19 . 














2. P. 


20/ 










X . 




2.S. 


21 


r 








X 




3.D001 


22 




X 




X- 






3.D200 


23 




X 




X 






3.D400 


24 














3.G. 


• 25 , , 








•< 


1 


X 


3.H. ' 


..26 


X' 








■J 


X 


3.1. 


27 . • 














. 3.J. 


28 














3.M.2, 


29 


X 


X 








X 


3'.N. ^ 


^ '.-..•30 














3.N.. • 


■ 31- 














3.E.I. 


32. 












~ X 



' ' ' '1 

For a variety of reasons, different states* case files did not contain; 
the type of information defined. as corroborating an R-300 item* 
Consistent codes were assigned in these instances. . • . 
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• There would be an increase in the accuracy of the data sub- 
mitted tO'RSA. 

• The procedure would help to, identify problem areas within the 
agency, 

• • The^ prc?cedure would improve purrent .data processing systems. 

• The ^procedure would provide /greater accountability i-n terms' 
of client expenditures, status movement, cases" documentation, 
and the use of medical consultation and medical reports. 

• The proced'ure would establjtfsh some uniformity with fespect to . 
the nationwide collection of data from rehabilitation agencies. 

• The procedure will help agpcy personnel gain insight into 
casework, provide a betteil understanding of the R-300, and 
field problems. I 

• The procedure might resiul|t in better organized case folders 
and uniformity in reporting procedures, ^ 

evaluating the forms being used and 
edures for completion are being 
cts . 

Thrust of' Instrument 

.Based on an overall evaluation of the prop'^sed procedure, most states 
agreed that the goals set by Standard 9 are critical to the delivery of 
quality services. The R-300 Verification is a good check of the reliability 
and completeness of information submi^tted to RSA. On the other hand, some > 
states pointed out that verification of information does not take "validity" 
into account, and that the. definition of verification might be broadened to 
analyze the flow of information through all stages, not just the information 
on the R-30Q versus the supportive evidence- ' 

General Format | * 

• There were no major dif filicultics found in the format of the R-300 
Verification instrument. Most found it clear, easy to read, and of a 
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reasonable length. Tfie data collection process and answer scale were 
structured sufficiently to allow reviewers to make comfortable judgments, 
_ but there was some ambiguity about the missing^' category. 

Reviewer Qualifications and Training * ^ ' 

Perhaps one of the areas where the greatest amount of concern was 
expressed w,as the qualification of the reviewers, and the training required 
for inter-reviewer reliability. There was agreement that'the verification 
procedure should be done only by .people 'familiar with case files, forms, and 
the VR process, but no oae seeme^ to agree exactly who met these criteria. 
To correct any discrepancies among those chosen as reviewers., there was 
agreement that a comprehensive training format is needed. This training 
would include reviewing a dummy case to assure understanding of interpreta- 
tion of items, and advance listing of possible places in the case file where 
verification data could be found. As for a specific training need, there 
were suggestions that extra attention be paid to SSI and SSDI cases, as well 
as cases involving psychotic or psychoneurotic disabilities.' 

Summary j A 

The previous discussion has raised several issues with respect to the 
R-500 Veri^irtHHjj^^i^^^ that need to be 'addressed. The R-300 Verifi- 

cation IS primarily desigiied to serve RSA's need for quality data. Toward • 
this end, we recommend that there be particular R-300 items that .>each agency 
must verify periodically. To ensure that each state can verify these items^^ 
some of the data ayailability problems noted earlier must be corrected; in 
some cases states may need to change their data collection systems. For 
example, in states where other agencies have responsibility for maintaining 
certain data items, the VR agency should arrange to receive copies of the. 
back-up documentation. We feel that, in general, the changes required are 
justified by the benefits received from reliable, accurate R-300 data. The 
actual verification procedure seems to be relatively inexpensive and not 
overl) time-consuming, considering the increase in the validity of data 
analysis made possible through implementation of th'is procedure. 
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Results of Protest Data 

Tables 50 and 51 present the results of the R-33t) Verification for"^ 
referral and closure information. As was noted earlier in the discussion 
*of .this standard, there are many items oh the R-Z(jO that states found to 
be unverifiable under existing data collection systems, JPtiese i^verifiable 
items are presented in T^ble 49, Taking these into consideration, some of 
the lower performahce^evels and consiste^cy ratios shown oft Tables 50 and 
51 may be explained by the tact that the state(s) considered the items to 
be unverifiable* In an implemented system, these data availability "problems 
would need to be rectified. On the other hand, some supposedly unverifiable 
items s,howed high performance levels with a high degree of consistency among 
reviewers. We do not know the reasons for this apparent inconsistency. 

These problems with differing data collection systems limit the* inter- 
pretation of results to those items considered to be verifiable and that 
show an acceptably high degree of inter-reviewer consistency across states* ^ 
On th,e positive side, most items collected at referral showed fiigh perform- 
ance levels and consistency ratios. Tliese items include basic infor- 
mation such as a client's age, Social Security number, primary disability 
data, work status, and earnings* For closure data, verifying work status at 
closure and outcome status showed particularly iiigh performance levels and 
consistency ratios. On the negative side, verifying referral data suth as 
secondary disability information, previous closure infaj?mation, and primary 
source of support showed somewhat lower perfc)rmam>e levels. For closure data 
verifying the reason for non-rehabilitated closure showed t:onsistently low 
performance levels (mostly below 50%). 




•fab^ SO 

R-300 Verifications 

(Percent of RevicKed Cases for tVhich R^50Q Inforroat'lori 
is Consistent with that Found m the Client's Case Recor<j) 

Referral. Data 



' ■ "yA 


. DE 


MI 


MS 


OR 




VA 


Age 


I 

♦Performance {%) 


98 


100 


95 


96 


99 


92 


* Consistency Ratio 


4 

1.00 


1,00 


.95 


1,00 


1.00 


.98 


Social security 
ntmioer ' 


Performance {%) 


4 

100 


*100 


95 


98 


'99 


97 


Consistency R^tio 


1.00 


l.QO 


,98 


1.00 


1,00 


.98 


SSDI status at 
referral 


Performance {%) 


94 


k 


97 


/ 81 


93 


7 


Consistency Ratio 


1.00 


KOO 


,98 


,79 


1,00 


.93 « 


SSI status at ' ^ 
referral 


Performance f'i) 


86 


9S 


94 


82 


9J 


6 


Consistehcy Ratio 


1.00 


UOO 


1,00 


,75 


,98 


,93 


<? 

^lajor (primaryj 
disablini; condition 


Performance f*#) 


88 . 


91 


98 


97 


92 


100 


Consistency Ratio 


1.00 


,98 


1,00 


1.00 


,90 


,95 


The disability code 
for major disabling 
condition ♦ 


.Performance {%) 


84 


85 


' 3 


94 


9: 


99 


Consistency Ratio 


i.oo 


,94 


1.00 


.92 


.92 


' .91 


Secondary disability 


Performance {%) 


67 


83 


86 


88 


84 


' 97 


Consistency Ratio 


.92 


.82 


,91 . 


.•96 


,87 


,91 


• 7, ' 

The disabil ity code 
for secondary dis- 
ability 


Performance (*) 


•67 


80 


87 . 


' 93 


81 


. ^7 


Consistency Rati§ 


.92* 


^ .76 


. ,86 


,96 


.90' 


.91 


Previous closure of 
the client within t 
36 months*^ 


Performance {%) ^ 


2 


:91 


17 


75^ 


^ '91 „ 


5 


Consistency Ratio 


1.00 


1.00 - 


,91 


.88 


.88 - 


.60 


Outcome irefKibili- 
tated or no^ rehabil- 


RcrrSrniance {%) 


2 


21 


11 


10 


13 


3 


itated) of the clienjC 
of pravioub closure 


Consistency Ratio 




.94 


,57 . 


,75 


• .73 


.71 




Performance (*#) 


96 


92 


93 


..98 ~ 


\^ 95 ' 


94 , 


^ork status at 
referral 


■ r 

Consistjpncy Ratio 


1.00 

— ' — ■ * 


. .94 


,95 


.96 


i 

,98 


« 

.88 


'■ . . ' 


Performance f*#) 




-92 . 


94 


97 


* 83 


91 


*'ecrkly C'irnuijir at • ^ 
referral ^ 


Consistency Ratio' 


.92 


.89 


,95 


,92 


.75 


.79 . 





^iy'yi|ance {%) 


* — Y — |, 
82 






. 90 


J " 

^ 84 


93 


Pi' inary source ot ■} 
support- at'referr^ 


wjjf^^tcncv Ratio 


.92* 


.83 


. <81 


,92 


.85 


.86 




120 


li9 


119 


100 


98 


118 ' 
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Table 51 
R-5QQ Verifications 
(Percent of Reviewed Cases for Which R-300 Information 



is Consistent with that Found m the Client's Case Record) 
Closure Data ' 





DE 


MI 


MS 


OR 


PA 


VA 


Identification of 
client as TF at 
closure^ 


Performance (•) 


94 


1 


99 


N'A 


91 


14 


Consistency Ratio 


1.00 


NA 


1.00 


NA 


.92 


.86 . 


Identification of 
client as SF- at * 
closure^ 


Perforraance (%) 


98 


NA 


100 


NA 


92 


9 


Consistency Ratio 


1 .00 


NA 


1.00 . 


p ~ 

NA 


.93 


,86 


Identification of 
client as severely 
disabled at closure^ 


Performance 


.92 


90 


97 


' 78 


87 


90 


Consistency Ratio 


1.00 


.98 


.95 


.75 


.87 


.93^ 


SSDI status^t 
closure 


Performance (%) 


98 


92 




58 


90 


2 


Consistency Ratio 


1.00 


'.93 




•,63 


.85 


.86 


SSI status at 
closure 


Performance (*•} 


94* 


92 


83 


59 


87 


S 


tonsistencv Ratio 


1.00 


1.00 


.83 


.58 


.O" 


.86 


Work status at 
closure *' 


Performance (%) 


100 


98 - 


95 


92 


90 


95 


Consistency Ratio 


1.00 


1.00 


.93 


1.00 


.95 


1.00 


Weekly earamgs at 
closure 


Performance (%j 


100- 


^ 92 


83 


86 ♦ 


78 


91 


Consistency Ratio 


1.00 


l.&O 


.91 


.96 


!85 


.91 


7 he number of nontns 
from client accep- 
tance to closure 
fstatus 10-24) 


Performance (%) 


100 


,100 


96 


93 




0 


Consistency Ratio 




1.00 


1.00 


KOO 


.98"^ 


.86 




Performance (%) 


.100 


100 


97 


. 100 


94 


'/ 

•99 


Outcome status 


Consistency Ratio 


1^00 * 


1. 00 


.98 


1.00 


" 1.00 


.98 


If closed 08, 2S, or 


Performance (%) 


51 


48 


26 


44 


49 


41. 


30,' reason for non- 
rehabi 1 itated clOMirc 


Consistency Ratio 


1.00 


.96 


1.00 


1.00 


.?)7 


.9; 


The tot J I LOSt or 
all case services 


Performance (%) 


94 


^ NA 


64 


1^ 


9 • 

87 


^0 


(within a 10\ nnrgin 
of error) 


Consistency Ratio ' 


^2 


NA 


.93 


NA 


.85 


NA 


N =» 


120 


119 


119 


100 


98 


US 



.Social Security Trust Fund, on Federal Special Program I.D. 

^^Suyplcnfenf.T I Security Income Funds, on Federal Special Piogram I^. D. 

c * ' ^ 

On Federal Special Program I.D. 
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TIMELINESS ASSESSMENT 

The Timeliness Assessment responds directly to Standard 11. It is 
designed to link subjective assessments of the timeliness of case movement 
to objective data on the length of time spent in various statuses by 
different, disability types. This will generate a data pool from which 
one can draw parameters for time-in-status , An exalnple of the Jimeli- • 
.ness Assessment can be seen in Exhibit 11.- * 

The procedure for pretesting the instrument was as follows. The 
reviewers recorded their assessments of a case's timeliness of movement 
through the VR process. Separate assessments are made for the referral 
and acceptance stage (statuses 00-08/10) , for the plan development stage 
(statuses 10-12) and for the service process (statuses 12-26/28/30). 
These assessments are subjective, in the sense that no specific criteria 

(i.e., times-in-status) are applied to determine timeliness. Rather^ the 

ft 

assessments require jud^mentsf. regarding the appropriateness of the speed 
of case movement, given the various circumstances impacting on a particu- 
lar client/s caseflow. Reviewers need to be thoroughly familiar with the 
case circumstances before attempting to assess timeliness. Familiarity 
was gained completing th'e CRS, the R-300 Verification ilistrument , or by 
conducting an independent review of the case file-. 

The. same cases receiving Timeliness AssessmShts were alse- being re- 
viewed by the SDSU CRS. Section I .A. of that instiSument Cidentifying 
information) recorded the objective data (date of r^eferral, date of eli- 
gibility determination, date of IWRP completion, Hdte of Closure, client's 
disability, etc.) needed to link these assessments with other client data 
in arder to develop case flagging analyses. It was not expected that the 
MEU's would generate a Sufficiently large data base to establish timeli- 
ness parameters^ but rather they would demonstrate whether professionals, 
with casework knowledge , 'could reliably assess the timeliness of a case's 
movement, the sufficiency of documentation in instances of undue delay, 
and the causers *for that delay. 
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EXHIBIT 11 



I 



"J 3 3 6 



3 9 10 U 12 



Sccric.v VII; t:.'2l:ncSS rhsharoi v.'o cc;:2>rr3 

A. TI.*2i:S'£SS 



TIME STARTcO 



..iSiAL'Cr,o.SS- AS incucacors oi :i::2Ur.css or :he /arious phiies m cnc rehaoiiicacion process, consider where 
appiicao.e :.^.c in;or=aciQn Iistea below pertaining co each of the status mcrenencs xn coluais A. 3, and C 
wnen responcinj to quescicns 1-3. ' 



ERIC 



Status 00-C3/10 
Referral co 
Elxjibi lit/ 
Decision 



T:=e lapses becveen referral of :he client and the iniciai client inter/iew, the 
request :or information, the receipt of mfomation on the client, additional diag- 
nostic ser/icss, e;jtended evaluation (if appUcable)^ and any jaos between or during 
these events and the final eligibility decision and notification'. 



Cli- 



cks Section I, Iteas 11, 18, and 22, 



3, Status 10-12 Tir.0 lapses between eligibility determination an<£ IWRP approval, anount and frequency 

(ilig:bii::y to or client contact during plan develcoment. 
Plan Approval) ?*r>-*?r^- Section C, Ueas 13 and 19. 
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Training and Logistics 

Training. for the Timeliness Assessments took place in each state 
either .as an adjunct to the. Case Review procedure (Oregon and Pennsylvania), 
to the R-300 Verification procedure (Delaware, Mississippi, and Virginia),, 
or^ as an independent activity (Michigan) • *^ - 4 

Issues Raised During the Pretext ' . . . < 

V 

Cost and Effort 

No MEU expressed any real concern about the cost or effort involved 
in conducting the Timeliness Assessment. IVhen an MEU expressed concern 
about the Timeliness Assessment, it was usually in relation to the design 
^f the procedure, or some special circumstance peculiar to the agency 
thkt might distort the results of the assessment. 

Comparison to State Methods 

Most MEU's either lacked -case-flSgging procedures, or have found 
their present systems to be inadequate. The greatest problem with existing 
systems seems to be the divergence of opinion between counselors, field 
administrators, and program evaluators as to what constitutes untimely ser- - 
vice. Some systems are biased on administrative standards of timely case 
movement, and some are based on normative standards of case movement. One 
^>1EU pointed to particular problems encountered in using its fixesd time-in- ' 
status system (the system provides a semiannual printout which flags cases 
staying, in particular statuses too long). The reviewer noted that, the 
system simply created more work for counselors, and usually provided infer- 
m'ation too late to be useful (c'lients had often moved on to later statuses 
before the counselor was alerted to the "problem'*) • 



Possible Benefits 

With, the ultimate goal being the development of a workable case- 
flagging procedure, most MEU's found thq. Timeliness Assessment to be an 



^apptopriate and needed Procedural Standard* Sotae possible benefits of . 
.implementation of this procedure include: 

More attention will be paid to clients staying in a particular, 
status too long; . 1 * ' - . 

• The Timeliness A^s^ssment will aid greatly. in pinpoi-nting 

• 'problems in service delivery;, " " . 

^ ' ' - • The procedure may result in an increase in successful rehabil- 
itations, ^as some MEU's believe efficient case -jiipvement is 
related to outcome; \ 

• The procedure- may improve documentation' of time-in-status 
requrred to complete the VR process. 

I Tnrustofthe Standard and Instrument 

Some MEU's.were concerned that the system, as it is now designed^ 
seems to define ''untimely*' 'as '*too slow,'* and seems to ignore cases where, 
/'untimely'* movement results from the client being rushed through some 
stages of the system. If this system is implemented^ reviewers felt ' 
that pressure for timely service may result in counselors attempting to 
speed client* movement through the various statuses* We consider thig'to t 
be d valid concern. The .instrument must be revised to ^allow identification 
of too-fast movement, as well as movement which is too slow. 

Another concern expressed about the Timeliness Assessment was that 
it might fail to function effectively when an agency, due to cieclining 
resources, implements a waiting list. Use of a waiting list is exp^ct^d 
to increase the time spent in status 02, and thu^ to distort assessments 
of timeliness during the referral/acceptance process. We agree that this 
presents a potential problem. During training; all reviewers nee^, to be 
made aware of the potential for this problem. It should* be noted toat 
item 5a "lack of resources" does alloi^?- for ai^ explanation by review- 
ers. However, given the potential for widespread occurrence of this prob- 
lem, it might be better^ to not^ code, such cases as timeliness problems when 
it^ is clearly established' that the: client was on a^^ waiting Ust. * ; 
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format • v. ■ • 

' I . *« 

For the most part, the NiEU*s^ found sectipni A and 'B of the ^instrument 
* < > 

' easy to assess. Section C, however, presented more of e problem,, as this 

section; requires more judgment and does not usually contain ^'obvious" 

delays. 

A3 no*ted earlier, reviewers were -prim^^ily conq^med with the sub- 
jectivity of the timeline^s^decision, but tliiiB- concern was' further height- 
ened by the format of the instrument , Itself^. For Example, when attempting 
.to identify the causes for untimely case movement, frequency of counselor/ 
client contact can be misleading: quality of contact (i.e., information ex- 
changed and progress made) is .more important -than quantity of contact. Also, 
the "cauiJI of d.elay" (items 3a-3i) were thought to. 'Contain some overlap- 
■ pjng categories, and in ^some cases the documentation may not give a cl^ear 

: indication of- all actual causes for delay. For exajnple, /'lack of counselor 

coTjtact" may be due to *4ack of resources" or ^'administrative de-lays." 
> Our response- to this -last problem is as follows. 'Ideally,- with sufficient 
training and. reviewer familiarity with the VR service process, and with 
improved dociimentation (resulting from, implementation of the Pr.ocedural 

^Standards}, it is hoped that reviewers \Vill actuaMy be. able to identify . 
^s many causes for delay as apply to a particular case. This will allow 

' greater understandings of common simultaneous problems,* and of the root 
(i.e., main cause) of^those problems,' This information .could be used t:o^ 
smooth the case flow and improve agency functioning. ^ . • . 

TraiTiing ' * ^ i * . " ' ' " 

To minimize the' subjectivity of Timeliness Assessments, all MEU's 
agreed that exta^^ive training of qXialiTied reviewers is needed. This 
' training must include teaching consj.stent definitions of t^rms and materials 
to. help standardise interpretations, and taking, time to thoroughly analyze 
example cases to make the best possible, though subjective ,t assessment of 
•'timeliness*. - * . 
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Sufnuna-ry and Response 

The majx)r con^^'rns of reviewers related to the inherently subjective' 
nature of the Time|l^ess Assessments, and to the failure of the form to 
allpw for explicit Identification of- cases which moved too fast. We have 
already noted the need to revise the form to correct the second problem. 
With'fegard to the subjectivity problem, we would note that the problem 
will always exist to some extent .v However, we feel that uniformity of 
^interpretation by reviewers can be maximize'd by using the following pro- 
cedures. First, provide adequate training on agency and federal terms, 
on source documents,' and on current best practice. Second, choose review- 
ers from amonS, staff witfPaa^tfi^te experience in casework and, hopefully, 
in casework supervision and agencyN^rations, These people will have a 
head-start in terms of familiarity with" definitions and documents, and 
tYiey should already ^be familiar with '*best practice*^ (even if not the most 
current best, practice) • Third^, conduct "dry runs, and haye reviewers dis- 
cuss commonly reviewed (i.e., reliability) cases, before the data collec- 
tion process proceeds too far. This should serve to point .out , potential 
consistency, (and subjectivity) problems in advance. Finally, use the s.ame 
'reviewer to collect all the' Procedural Standards 'Sata for a particular 
case, and have the reviewer conduct the data collection in the following 
order: / 

1) R-300 Vprifitation 

2) CRS or MCRS (Eligibility and^IWRP) 

3) Timeliness Assessment. ' n 

By following this ofdfer, the. reviewer is forced to digest as mych 
•'objective" background information as possible, and should have an under- 
standing pf the case which is exceeded only by the .counselor and po3sibly 
by the, relevant counselor supervisor. This level .of "objective" knowledge 
.tnd imJerstanJing can only help the r9viewer in making- the required "sub- 
jcctivc'^ tineliness <^ssessment. . , ^ ^' . . . ' 
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Results of Pretest Data 

Timeliness in Acceptance Stage ^ 

Table 52 pnesents the results of the Timeliness Assessment for cas^s 
in the acceptance phase, covering movement through statuses 00-08/10. 
shown by the table, the percentage of cases served in a timely manner 
each MEU ranged from 59%-95%. Corresponding consistency ratios for thfese 
assessment's were generally over the .750 required for a valid results, with 
one exception. These results would seem to indicate that for the most part 
casefiles contained the documentation necessary to make a Timeliness Assess 
ment,' and in most ceases the client moved through the acceptance phase in a 
timely manner. 

Por tho^e cases not served in a timely manner, casefile documentation 
showed potential reasons fo? the delay in 17%-80% of the cases. Among 
those reasons most frequently mentioned were: 

• No response. by client . ^ ■ 

• No contact/action by counselor / 

• Delay in receiving reports 

^ . ' J 

Timeliness in Plan Development Stage 

Table, 53pr^sents the results of the Timeliness' Assessment for cases . 
in the plan development stage, covering movement through statuses 10-12. 
The percentage of cases served an a timely manner through these statuses 
wpre all over 80%, with all consistency ratios above .750. These results 
are somewhat higher than for the eecceptance pha^e, showing that once 
accepted into the system, clientil move at an appropriate pace into actual 
services. , . ^ 

For those cases not receiving timely service in the plan development 
stage, casefile docum^tation gave potential reasons for the delay "in 
31i-100o of the cases. Among those reasons most frequent l^given were? 

• No. response by client ' * 

• Client unavailability 
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Table 52 
Timeliness Assessment 
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Table 53 - ' - ' ^ ■ 
TimeliTvess Assessment 
' 10-12 Plan Development 
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Timeliness in Service Provision Stage 

Table 54 presents the results of the Timeliness Assessment for cases 
in the service provision stage, covering movemen"*? through statuses 12-26/ 
28/30. The percentage of cases served in a timely manner through th^se 
statuses ranged from 42%-94%, with consist^en<5y ratios of between ,50 to ,88. 
these results are somewhat lower than for thep^evious tables, showing that 

^some clients are bogged down in these statuses, and that in many cases 
the casefijes possibly do not contain, the documentatibn needed *to make a 
Timeliness Assessment'. 'The subjettivity "issue no douftt-also is relevant 
here . ■ . ' \ : 

For those cases not served in a> timely manner /casefile dociimenta- 
tion gave a potential reason for the delay in 48%-65% of the cases. Among 

those reasons most * frequent ly given were: . ^ * * 

• No response by client 

• Client i^ndecision 

• ClieVit unavailability - . ' 
'\ • No contact by counselor 

-% No action by counselor . . * 

General Observations abj^ut the Timelinesj^ Assessment ^Data 

The*three previous discussi-ons have presented the results of the 
Timeliness Assessjnent for the MEU's, covering movement through all statuses 
of tbe VR process » From th^se discussions, some general observations can^^e 
made about timeliness assessments . First, while more data from mpre states'A^ 
is* needed before performance levels can be interpreted with confidence, 
the^ results of the pretest suggest that current documentation procedures 
are not complete enough to allow for a clear Timeliness Assessment: This 
is reflected by the ji^w consistency^ ratios shown for various stages*©? the 
VR process. Also, cases that have not been served in a timely manner do 
not contain adequate ^documentation of the possible reasons for the delay. 
This is reficQted by the consistently low percentage of untimely cases 
where thc^ reason for the delay is given. If the .agency is to improve the 
timeX- delivery of services, it must have an indication of what is causing 
the delays. , - ' ■ . .J 
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Table 54 

Timeliness Assessment 
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A Model for Case Timeliness Monitoring 

The MEU pretest allowed exploration of timeliness in six state agencies 
Because this issue is recurrent in the discussion of standards, and has 
been addressed by RSA in the existing General Standards', RSA. and BPA felt 
that further investigation was warranted. Therefore, in addition, to the 
MEU data collection in the pretest, BPA sent, a request to all state agencies 
for documentation on their current case flagging procedures and on other,, 
more general, quality assurance approaches. We provide a detailed analysis 
of the findings of both an analysis of the MEU pretesjt data arid a review 
of current state case flagging systems ni" another report."^ Here, we summar- 
ize BPA's recommendations from that report, including a model for state case 
flagging systems which integrates state concern, for excess time in process 
with the use timeliness assessment procedures, 

BPA reported in the review of existing state timeliness systems that 
there are numerous approaches to monitoring cases for undue delay. The 
selection of a ''be-st approach" is not currently possibJLe,- since our analysis 
showed a variety of stat^ conditions and client characteristics that .may 
affect both the time in status itself and a rater *s own assessment of the 
"timeliness" of a particular case. IVhile the MEU/^ndards pretest data 
sheds some light on the relationships of timeliness and time in process to 
outcome of casfes,'the limited' sample and measurement issues preclude our 
ability to project the findings for all state VR agencies or even classes 
of agenci-es. However^ this investigation does indicate that a concern for 
timeliness is important both for assuring- the quality of case services, and 
for providing the best chance for successful outcomes of these services. 

Based on the analysis, BPA recommends: . i^- 

A* ^ ^ 

♦ Tliat any case flagging procedure recommended by» RSA must be 
reliable across states. Trained 'ratcr,s should, apply the same 
procedures and specific criteria across states if the find- 
ings are to have validity and reliability. Such an approach 
argues for' assessments done by RSA at the regionali* Level or 
l?y a national as.sessm^nt, project rather than by states' indiv- 

- idual staffs ... 



" ^ Review of State VR Agency Procedures f^WIase Flagging and Quality 
Assurance (September, 1981) >\ " 
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• The procedures should focus pn identification of untimely 
I^ractic^, over/ll, -by type of plan,'-^ types of client, 

and by status. This identification -^should be developed . *' 
, from data analysis on improved 'measures of timeliness and ' 
improved documentation of times in status on the R-300. 

• Kith improved measures, mora* analysis^ can be done on the 
relationships between ^times in status and timeliness assess- 
ments, and*^ outcome data from the R-300. Such analysis will 
lead to identification of norms for case flagging, aftd pro- 
vidd a component of the supportive evaluatio/i ^or 'decision 
support system for RSA MIS.i 

Tliis, our long-run recommendation is that timeline^s/measures and 



procedures must be improved and be more uniformly applied to a large. 

* ^ 

#^ number of case:^ before an empirical basis for setting timeliness norms^ is 
possible. Ke see tliis developmetital work as becoMng more possible through 
•thq MIS, if tineliness assessments and status , times are included in the ^ 
data elements and if the uniform assessment practices^ are followed in col-* 
lectins the data. 

In the short run, we suggest the following model for iipprovement of . , 
case flagging practi.ce and use of time in status, standards at the state • 
level. .A. St ate should not flag too many cases, because, such flagging . . 
would b^ inefficient. However, flagging too few cases will possibly leave 
too 5)m>ny untimely cases in the system without examination. The model, 
therefore, is based on examining the number of cases being flagged with 

* the states' existing 'flagging standards, in relationship to an analysis 

• Qf the service process and the overall caseload, and refining state fl^ag- 
ging s^ancjards . * , * . 

Figure 6 shows the' steps to follow in the analysis of undue delay in , 
the»caseload. A case is Selected for review, following the sampling design 
used by the statX agency fo!r timeliness review (100% of all cases, or .state 
random samplii^ procedure).* State timeliness standards (allowed times in 
process) ufU bb'used in this model. The model calls for upwards or >dpwn- 
waVds adjusting of these times in process standards (increases or decreases 
in the maximum time allowed in each status) based on two additional stand*- 
ards for the caseload itself: 
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Figure 6 : Model Case Flagging Systera 
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•1. Not more* than 20% of fagged cases should be' timely.^ If 
more' than 20% of ^the flagged daSes are judged as timely fj . 
when they are reviewed, the system is flagging cases un«- 
necessarily, and^the fla^ggiijg standards should be less 
stringent (times allowed in the statuses coulas^be increased)., 
2. Assuming the* conditions in (1) above hold, at least 5% but 
not more than ,10% of all cases should be flagged. If su'ch 
excessive flagging occurs, and the flagging represents cases 
judged untimely, 'then there is a problem with the service 
delivery system itself, and an analysis of the process is 
-^'v^ • called for. If less than 5% of cases are flagged, the flag- 

' . ging system should be more stringent (timest allowed in - 

- statuses should be decreased). 

Tlie model in Figure 6 uses both .quantitative information on times in 
I ' * 1 

status and subiective infomfeiion on timeliness to decide about cases, 

-In the first stage, a case is selected for attention. If time in -process 
is all right; for the case, i-t ts returned to the file. If time in process 
exceeds agency standards, tlie case is flagged and reviewed as to the time^ 
lincss of process. As we h\ve pointed out, while there .is a relationship 
between timeliness and time in process, it is not a one-to-one relation-/ 
ship, so it is possible that flagged cdses will be judged timely. If so^ 
they ^Iso can be returned to the file. For both these groups returned to 
the file (labeled A and B on our diagram), the number of, such cases should 
be recorded. Likewise, the number of untimely cases flagged should be 
counted. ' (In' our diagram, this is C") -Cases should be flagged and re- 
viewed until the planned sample size (A + B +*C) is achieved. ' ^ ' 
. . Once the sample is complete, Ithe system asks three questions of the 
ca^cs. 'First, do the timely cases (B) exceed 20% of all flagged (B + C) 
cases? If yes*, the system may te flawing too many cases, and times 
allowed for each status could be increased. If, however, B/CB + C) is 
less than 20%, ^e ask whether less than 10% of the total caseload was 
flagged. If so,„then the time in process standards appear in equilibrium 



1 . V • ' 

Tlie model is, however, concerned only- with times-in-status which are 

too long ; . It doe^ not address ^'rushed^' cases!. Cases handled with »'undue 
speed" are a separate issue* requiring special state attention. 

< • . ■ . ■ , • • 

384 



358 . 



for the state (hot too many cases are being flagged; of the cases that are 
flagged, most of them are indeed untimely cases), ' If, on the other hand,- 
more than 10% of th'e cases are flagged, there is a problem in the service 

process itself, since these cases* have been judged as untimely and there 

. ^ ' C V 

rv. are too many untiipely case-^ for effiicient moiTitoring and efficient operas 

tions, .This- calls for an examination of the service process itself, 
* perhaps* using the* decision support system to analyze the state caseload , 
process and pinpoint timeliness issues in *relationshit/ to client outcomes ^ 
and costs. In addition, this problem may call for upward adjustment of 
the times allowed in statuses, to flag. fewer cases. However, checking for 
the appropriate times must be done in 'knother iteration so that a check 
. can be made as to whether both the 20% an^ 191% standatds are met for. a 
given new standards level. ' 

Finally, .the state^should routinely flag between 5%^ and 1Q% of its 
cases, to assure that flagging standards are set low enough. If less than J 
S% of cases are flagged, the standards should be made more stringent 
(al Lowed tijnes in status decreased) before the next round of review. 

Using .this approach^ states can adjust their times in status standards * 
upwards or downwards' to be more meaningful and to result in an efficient 
, process that spots problematic cases without excessive monitoring. 
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'fx. ' summary and recommendations ^ 

^ FOR XHE PROCEDURAL STANDARDS' * 

, In previous sections we^iiiaVe discussed the forn ,of and ifitionale for 
the proposed Procedural Standards ? the data collection instruments proposed 
for implementing Jthe Standards; the pretest -proces?; and t>he findings 
obtained from the pretest data. As welL, we synthesized the'* cbmments • 
received from pretest participants. In this section our purpose is to 
summarize the participants' comments into a coherent statement of hdw 
reviewers perceived the overall Procedural 'Standards ^pretest effort -7 that 
is, what w,as good and what was bad and to use the understanding gained 
froV the pretest to develop an improved Procedural Standards system design/ 
suitable for n^tionwidel" implementation. 

Beginning with the^positive, reviewers in general felt that the pro- 
posed Procedural Standards are appropriate for inclusion in the overall VR 
evaluation system / We feel that thi^ is a correct asses^]fient. (Jbhs ideAng. 
each of the Standards in turn, we would assert that: 

• RSA needs assurancg , that the basic data it receives from states on 
clients served is valid, accurate^ and reliable, ^s required by Standard 9. 

I Data validity is important since the data is used by RSA both for repor,ting 
to Congress, and for performing program and ^licy analyses intended to 
help improve th,e program. > . ' 

' • State agencies must follow federal regulations regarding eligibility 
and provide adequate documentation hi. the basis for eligibility decisions, 
as required by Standard 10. This is important so that state agencies, 
thorough their ''gate-keeping*' function, may reserVe servi(;.es for those 
individuals. meeting the criteria for need established by Congress. ^ 
extension, this issue is important due to its impact on an agency's cover- 
age and on its fundamental cost-effectiveness.' 

• Agencies should endeavor to ensure that clients move through the 
process at a pace which is "appropriate to their needs and capabilities,'* 

ds required by Standard 11. We feel this concept is appropriate for a 

standard, since it is one of the areas of service quality. over whiclf" coun- 

selors and agencies can exercise the greatest control'. ^ . . , 

♦ • 
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Agencies should ensure that clients are provided IWRPs as appro- 
priate,, and that the HVRP provisions.Xe.g. ! client participation ^and due 
process) are adhefed to. This is j{rnportai}t both from a compliance stand- 
point, and imrel'ation to the service quality issu'es embodied in the IWRP 
legislation. . • » 

In short, we remain convinced that^ the Standards as stated are appro- 
priate ^or inclusion 'in VR's overall ^vaj^uation sys^tem. Further, we feel 
that our vifewpoint is ^supported by comments received from the pretest 
participants r . ^ ^ ' ^ ' ' * * ' t 

. The next major positive, point to note is that, overall, reviewers 
seemed to feel that the' proposed data collection instruments are appropriate 
for use in assessing state agency compliance with th;^ Procedural Standards . 
This conclusion must be tempered by an understanding of^ exactly what is 
meant by ''appropriate'\ Vs discussed earlier (and as will be discussed 
below) reviewers actually perceived various problems witfi^he instruments 

^ when consid ered within the environment of the Procedural Standards -pretest. 



However, in term^ of their appropriateness in ^addressing the concerns of 
the Procedural Standards, theJ instrumentp'actually received fairly high^ 
marks from reviewers. For example', the . Case Review Schedule was seen as 
providing a thorough assessment of compliance with both the eligibility and ' 
IWRP prpvisions of tlae ^Rehabilitation Act. True,'som^ sentiment was 
expressed that the CRS " may be more detailed, time-consuming, and critical' 
than wha^ is needed to evaluate compliance." However, we inteYpret this 
statement to mean "what is minimally^ needed to ^t i general picture of 
compliance." UTiile^RSA may, someday, decide that a generil picture is 
adequate, we believe that 'dbrrently RSA desires a more comprehensive pic- 
ture of the' extent to which states are complying with the full , intent of 
the^ relevant legislative mandates. The CftS provides just such a picture. 
Tlie pnly^ caveat we would add is this: if .the Stajidards are impl'emented, we 
'would .recommend that the Modified Case Review Schedule be used to reduce 
the overall time requirement sS » 

Similarly, both the R-56 0 Verifi cation a nd the Timeliness. Assessment 

, * ' : ^ ' — * 

instifuments were seen 'as capturing the intent of thfeir relevant Standards,* . 
overall. Two fairly significant problems were noted, ^ however. First, as 
regard^ the R-300 Verification, the point. was made that fact the 
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instrument fails to assess validity of R-300 data (i._e\ to validate that 
the R-300 reflects ''reality'^' Since it relies solely on written documen- 
tat ion, the instrument is useful only for checks* of accuracy vis^a-vis 
backup documentation. We agree with this ,asseasmlnt T Jo correct thi$ 
problem, we would suggest that as part of the data collection methodology, 
prc^cedufes be implemented for conducting reality checks on items considered 
key for reporting and analyrfs (e.g. , conduct employer surveys to validate 
closure OGcupation, and wageS) . GivAk their cost, these checks Ishoul^d be 
conducted infrequently, for exaraplp every three to six years'; buV they would 
go far toward lending confidence in the accuracy and validity of the major 
data system used ^Ji VR. * ) ■ . ^ 

The^ second problem noted by reviewers concerned the absence on the 
Timeliness Assessment of any method for identifying cases vhere the process 
moved tocy.f^st. As stated earlier, we feel this is a legitimate concern. 
Our guidance iJvaterials will include a revised Timeliness Assessment Instru- 
ment which addresses this concern. 
^ • 'In sum, we^ believe that, overall, our reviewers reached positively to 
the content of the data collection instruments in terms o£ theirr appropri- . 
ateness for assessing compliancy with the Standards.} In addition, the 
Veviewers noted at variety of incidental benefits which potentially would 
be derived from collecting the Procedural Standards data. Among^the bene- 
fits noted^re the fallowing: ' • 

• improved case records; ' ^ ^ 

greater understanding of scope' and intent of federal mandates; 
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greater attention to the concerns embodied in the Standards 

-->. . ' • ^ 

(thia o£| course is the b^sic intent of including Procedural 

0 , j ' ^ *^ ^ • 

Standards, in the evalUatjion system); •* , , 

• uniform assessment of performance; and *^ 

• better 3;?iderstanding (by reviewers) of agency operations, 
caseioad^prattjces , and documentation, procedures. 

* : . ^ * 



^;\e have omitted discussion of various n^^ative comments regarding 
instrument and question format and question ^wording. We consider most of 
the issues raised in these comments to be pf minimal significance and, in* 
general, misdirected. The. ^xm^vcis, and our responses > are presented in 

adapter X. . ' ' ' ' ^ 



Given the varipus points presented above on the pi^oposed Procedural 
Standards and data collection instruments, W feel justified in recommend- 
ing thjit the Procedural Standards and instruments be iihplemented in the « 
.general form used in \;iie pretes^t; that is, after the Timeliness Assessment? 
has been revised and provisibn has .been made for validity- checks of, R-300 
data. We are further encouraged in making this recommendation given our 
, reviewers' percep^ons of the appropriateness of the Standards and the aver 
all acceptance of the data collection\instruments . However, while we feel 
justified in making this b^asic recommendation regarding whether and how 
performance should be assess*ed, there were a variety of issues (i>e., prpb- 
lems) raised during the pretest which should be considered when devising 
detailed recommendations for Standards implementation. We briefly discuss 
these problems below. Following' this , we attempt to draw out, .the implica- 
tions of these eompents for an improved design of the Tr^cedural Standards 
system. • . . " 

SUM^URY OF PROBLEMS .NOTED DURING THE PRETEST 

The px^test participants noted a variety of problems -which they either: 
(1) experienced during the Conduct ^6f the pretest, or (2) anticipated , 
shoul'd the Procedujrarl Standards be implemented^ as an on-going activity. 
.These perceived problems can be grouped into five broad areas, as discussed 
below. • ^ 

• First and foremost^ the Procedural Standards were seen as .requiring 
an excessive level of effort in order to adequately assess performance. 
This perception applied especially to the CaSe Review Schedule. The central 
issue, of course, .^is the amount of staff time requijed to conduct the Pro^ 
cedural Standards data coilection. Additionally, participants anticipated 
problems resulting from the intrusion, of the data collection proc*ess irito^ 
the -time available for staff to^ perform their regular duties (e.g., coun- 
seling, case supervision, etc.). 4 

• Second, the pretest illustrated clearly- that some method must be 
developed "^to overcome problems in data collection resulting from the /unique 
aspects of each state . In particular, the CRS^ and the R-300 Verification 
instrument presume a uniformity across^ states which in'f^ct does ^ot exist. 
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Problems^ were encountered as a result of unique procedures within states 

in situations where the law allows states some latitude for poliqy 
implementation — 'Which conflict \tith what the CRS implies is "correct*'. 
As well, both the CRS and the R-300 Verification instrument, by their 
nature, fail fo. allow for differences across states in terms of the type 
of documentation which can be used (jor, is sufficient)- to establish com- 
pliance with .various federal regulations. The problem here, as we see it, 
results less from the instruments' alleged inadecjuacies than from inadequate 
advance ;^ork done prior to the data collection, which* could have established: 

how the state is different, and therefore (2) how to respond to the 
revifew questions, given the staters unique aspects, 

» At a mpre mundane level, reviewers also experienced some problems m 
^ocating certain data or documentation due to unique aspects of the docu- 
mentation procedures in states. For example, reviewers who were (apparently) 
unfamiliar with .the state's R-300 equivarlent had some difficulty making the 
necessary correspondence*^ between the R-300 equivalent and the R-300 Verifi- 
cation instrument. As well, certain data was missing from case files 
altogether, not because of counselor negligence, but because of inter-agency 
divisions of resrponsibility for maintaining certain records or documentation ♦ 
For example, in som§ states, documentation of public assistance receipt may 
be maintained by state Departments of Welfare, These types of situations 
caused particular items on the instruments to be "not applicable," by def- 
inition. Further, if a review0r was unaware of such inter-agency arrange- 
ments, he/sh^ might mistakenly judge cases to be out gf compliance. 

• The thir4 problem concerns the subjectivity of judgements required 
by various review questions , * In the case of the R-300 Verification instru- 
ments, the subjectivity issue was related to the problems discussed above, 
pertaining to accessibil^ity (i.e., agency location) of documentation; and 
to advance knowledge needed for certain review questions regarding the 
documents which sufficiently established compliance. For the CRS, subjec- 
tive judgements we.re required both to judge whether or not* the documentation 
was sufficient to establish compliance, and (for 'certain questions) to judge 
the approp:piateness of services provided, decisions made, and so forth. 
Finally, the Timeliness^ As'sessmen^ required the most subjective judgemenlJS 
of the three instruments. Many reviewers found it difficult t^o judge the 
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appropriateness of case progress, and to determine the reasons for undue 

I 

delay* ' 

• A fourth problem noted by some reviewers was that they anticipated 
that "continual*^ training would be required if reviewers were to perform 
their function adequately and efficiently, once the Procedural Standards ^ 
are implemented. This issue arises, no doubt, directly as a result of the 
effort, ^'uniqueness,*' and subjectivity issues already discussed, 

• * Finally, a few reviewers expressed their belief that there woi/ld 

» 

be inadequate opportunity for feedback , and that such feedback as might be 
provided would be of little use in improving current operations. This 
perception seems to have resulted from two distinct factor^. First, 
revieweijs (Correctly) perceived the Procedural Standards as having a dis- 
tinct federal (i.e., RSA) focus. Given this focus, reviewers may simply 
have assumed that the Procedural Standards would be of little. use .to states. 
Sec^d, the preteat sample contained only closed cases. Byve^ccluding in- 
"service cases, ^ states are unable to intervene to correct any problems which 
may be occurring with the current caseload. ^ 

IMPLICATIONS OF OBSERVED PROBLEMS FOR IMPROVING THE PROCEDURAL STANDARDS ' 
SYSTEM DESIGN ' 

» 

Havi^hg provided an*overvie^v of the problem issues brr'^ought up during 

# 

the pretest, we will now attempt to respond to those concerns by recommend- 
ing sp.ecific approaches as regardiVarious aspects of the overall Procedural 
Standards system Resign. The questions to be addressed ipclude: 
- • How frequently should Procedural Standards data collection ' 
and reporting occur? • • 

• IvTiat sorts of criteria (in 'terms of qualifications and 
experience) should t/l useji in selecting reviewers? 

• {low can reviewer training be made most effective? 

, How can data collection be made most efficient, and how 
* can we enhance , the utility of reported data? • ^ 

• iVho shall hav-e^ r^esponsib^ity for data collection and , ' ^/ 
reporting,: RSA, or, the states? ' # ' , 

• Finally ,^ how shall we adapt the data collection to reflect 
variations in states' policies and procedures? 
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Frequency of Data Collection and Reporting 7;^--. - . . - 

^ We need to strike. a balance between "underreporting" of Procedural 
.Standards data (that is, allowing long time periods to elapse between the 
data collection/reporting cycles) and "oyerreporting" (e.g., requiring data- 
collection and reporting each year). There are. _pr 03 and cons to both 
approaches. On the one hand, given the importance of the Standards concepts, •» 
we might want to require faTrly frectuent assessments. of ^erforjiiance on the 
Standards* For example, an annual assessment would allow for virtually . ^ * . 
continual monitoring of .£rocedujres, practices, and dociimentation . » Additionally, 
if the full CRS is used, annual Procedural Standards data collection could ^ 
provide a^ rich additional data base for use in supportivj evaluation. 

On the other hand, it may be neither desirable, nor necessary to require 
annual data collection^ As* illustrated by the pretest, ^ the Procedural Stan- 
dards process requires a fair amount of effort. (Recarll that the CRS alone ^ 
required almost 44% of all staff time devoted to pretesting both the 
Performance Standards and the Procedural Standards). The^ overall level of 
effort required will presumably decrease after implementa<tion , since reviewer . 
efficiency should increase with case review. practi<:e and familiarity 
with the .instruments. However, the Procedural Standards will probably always ' 
require relatively high levels of effott, regardless of who collects* the data 
' (states or the feds) or how experienced theydat^ collectors are. Put simply, 
it is tine consuming to conduct the comprehensive* case reviews required by 1 
Procedural Standards; particularly given the need to sample sufficient cases, 
so as to allow for valid data analysis. ,The. benefits obtained may be out- 
weighed by the costs of requiring alri annual^ review. 

, Further, and perhaps more impoi^tant,. it simply may not be necessary to 



require annual data collection. Recall that the basic focus of the Procedural 
•Standards is on casework procedures, practices and docufnent^^tion^ related 
to compliance and quality service. Most likely, these aspects of state ^ , 
'agency operation da not change radically from year to year. ' Given this, 
the n<3ed for^ continual (i.-e., annua:!) ^s^essment" dil^^ears. However, 
this does not mean that a lengthy aissessment cycle- (e.g., every ten years) 
^ is acceptable eithpr. We still wish to conduct fairly regular assessments, 
SO ^as to 'encourage compliance, data accuracy, and attention to timely case 
handling. , • * 
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The solution, obviously, is to require assessment ^oiv the Procedural . 

Standards over a regular, but medium- term reporting^ cycle^ ; We feel that 

a t?hree-year data colJ ^ec^ipn cycle would be ideal, and that no more than 

~ ^ ^ 

four years should be allowed to elapse between assessments • Of course., we 

are speaking here of statewide assessnjents (i.e,, data collection and 

reporting for a sample of the staters total caseload, large enough to allow 

data analysis) • Spot-check assessments can (and sfiould) be conducted more 

frequently,, to monitor "problem" case workers and inexperienced caseworkers, 

and to monitor implementation of new procedures, practices, and methods of 

documentation.* Such spot checks are most appropriately conducted by , the 

state agencies, regardless of who bears" responsibility for the statewide 

data collection* Further, agencies should be free to use whatever method 

of review they deem acceptable:, the review methpd^s effectiveness will be 

proven or disproven by the results of the succeeding statewide assessment. 

Additionally, we would note that statewide assessments need not occur 
for all agencies in the same yea:^*. Rather, assessment ' could occur for one- 
third of the agencies each year, on a rotating basis. 

FinalTy> we would suggest that the* Vfeality checks," recommended as a 
supplement to the R-300 Verification procedure, be scheduled to occur in 
the same year statewide assessment (or as close to the assessment year as 
possible), so as to maximize the "interpretability" of data^results . However 
this may not be essential since as with^practices andf procedures the 
overall level of correspondence to "reality^Ws not likely to change 
dramatically from year to year.- • ^ > ^ 

* To summarize, we recommend tKat statewide assessments occur at least 
every three years, and no less frequently than every four years. Tliese state 
wide assessment^/shoiH^ be supplemented by, more frequent, ^targeted spot 
checks, as suggested by\the statewide assessments. Finally, ideally the 
R-300 reality checks will in the^same years^as the ' stat.evyide assessments. 

Reviewer Qualifications and Experience 

^ The Procedural Standards rest squarely on. -feh^ process of case review. 
The quality of those reviews, and their resulting data, is strongly 
influenced by the capabilities of the people who conduct the reviews. As 
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such, we warit -tci, utilize ti\ose personnel whcJ. are most qualified to review 
''cas?s/v Rather than attemp): •W.identif/'^e* existing §^Jf roles thin 

.'cJgenci^* or RSA, whicK might supply the mo^t appropriate reviewer personnel , ' 
we will instead identify th6 qualities we'would I'Sok for. in. reviewers, ^ Each 
*of*thp qualifications, n.oted below; are suggested in response^o problems' 
noted dujtin^the pretest. We wouftd note also ,thit our ^suggestions may 

.'aeeV, in tqtal, like'aVjlsh list" -for 'the /'super reviewer.". Potentially; . • 

'-;\t maiK Tiot't)e possible tA f ind aU the*desired •qualities' in combination. v. 

\Howeven -tt^. the extent weVan, ,the review process -will be-m^de more efficienri, 
validj, and 'reliable. We. seV the need -for Reviewers to sartisfying four* 
broad cfiteri^a; ideaLry^x±nvw)mbination: • ^ , • ' 

• The most important qu^ity desired in a reviewer is that^he^/sh'e 

^ have ansoAtima;?^ familiarity wiV a wide Variety' of aspects of st^te agency 



operations. Ea 



Each of three Procedural, Standai*ds data collection instruments, 
requires that reviewers possess detailed understanding of such aspects of ^ 
operation'^asf ^ v * \ • 



*casefile organization; ^ 

data collection forms and other documents maintained in cas^ 
files (e.g., R-300 equivalent, Vouchers, examination results^ 
IWRP'^s, etc.); " ^ ' \ 

, inter-agency arrangements for service provision; and inter 
agency divisions of responsibility for data collection and 
documentation; • . - ' . ' , 

agency policies regarding documents which are necessary and/or * 
sufficient *^to establish compliance, accuracy ,^ etc . ; 

general ager<^ "standard operating procedures" regarding client ' 
service flow, decision mechanisms , review procedures, and so forth; 

specific agency policies regarding target groups^and other priority 
:-elatO',1 concerns, as consistent witih'\federal law; . 
services available, for use by the ageincy,. inadequacies in 'overall 
service availability, and the^impacti of^service availability on 
service progress and quality; and 
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• agency terminology, vis-a-vis' federal' (e.g*, CRS) terminology, 
> * I ' 

In slK);rt, the ideal reviewer would have sufficient 'exj/eri^nce^with, 
and/or kapwledge pf the particuIaX agency, such tjjat probleip resulting ^ 
from agency idiosyn^crasies (policfies, itc.-fcan be avoided during data 

4 

Collection. ' t . ' - 

■ • In -addition, to having fami^liar^ty with the particular agency's 
^overall ope^^ioni, ideally ca^se reviewers would hav^ experience in casework 

(i.e., counselin-g) , casework supervision, and overall operations and 
" administration . Such experience will be valuable in terras of providing 
familiarity with procedures, doc^^ents, and .decision mechanisms. But -more 
important, it prdvides the, practical understanding required of reviewers when 
t^iey are. called upon'to make subjective jqdgements regarding the appropriateness 
of service^ pjovided,^ase progress, counselor decisions, and so forth.' - 
Such subjective judgements are easier to make and*are probably more valid 
and reliable -^'♦^yhen the reviewer can fall back on his/her casework 
experience, coupled with his/her awareness of broad agency goals and the • 
organizational 'and resource ^nstraints faced by counselors.- 



The third cfuality we^would look fjr is that of "aloofness**: the 



• reviewer should have no conflict of interest When reviewing cases. 
-* z — » ' ' * 

• Finally, our last criterion has less to do with the personal 
qualifications of reviewers th^n with the way^ we would like' to see the 
reviewer positioji structured. Specifically, we idealj.y would make the 
reviewing function one of the reviewers' main assigned responsibilities ^. 
In other words, do not shift the reviewing task to different personnel with 

each data collection cycle* The benefits of this are several. First, to the 

r 

extegit that Procedural ' Standards data collection is part of the person's over- 
all duties, we should be able 'to minimize perceptions that the Procedural 
Standards are intruding on the person's other re^lar duties. FurthW, by 
assigning the review .function to particular individuals, we will reinforce 
.^.the person's familiarity with the task, improve his or her. reviewing ability, 
^ and remov^ the necessity, for intensive training with each new d^ta tolleption 



Reviewer Training ^ , ^ ' ' * \ ^ 

\o a great extent;, the need ^or training should be minimized over .time, 
provided" that reviewe3>s possess the qualifications and experience o^the types 
discussed above.. However, there will still ^be need for training of some sort,* 
prior to each data jrollection activity, so as to assure maximum preparedness. 
This training should cov^r two bas.ic sets of issues • First, prior to any 
data collection, the reviewers n^ed to be reminded of ' the'^variousv idiosyn- 
crasies ibf' the^ particular agency which might cause 'problems in interpretation 
dt^ng J:he data oollectjLon, Most of -fthese gpints^wilL have been identified ' , 
in^eg.rlier as\etsment cycles; thus what is needed is to simply remind- the 
reviewers of the various points. Also, to the extent 'that new procedures 
or docun(entatioris have been implemented since the last cycle, the reviewers 
should be made awsyre of these changes. ' / \ * 

The second major activity which 'must occur during trai^ning is the set 
of procedures which are designed to assure maximum data validity. Specifically, 
the^onsistof : « - ^ . 

' • con4ucting reviews on /^^reliability cases*' (cases reviewed by all 

data collecfoTs) prior to initiation of the actual data collection 

procedure; ^ . ^ 

• computing the consistency ratios for each of the data items collected 
oh these reliability cases; ^ i.^ ' 

• having reviewers discuss these cases, ''both in terms of the overall 
picture of the. cases which emerged from the reviews, and in terms ^ 
of specific variables which proved difficult to assess reliably; 

• arriving at consistent decision-rutes for dealing with ambiguous ^ 
or problematic case circumstances; and ^ 

• doc.umenting these decisions atid rules, to allow other partl'^s x 
(e.g. RSA) to interpret data results. l| 

By conducting these twof broad training activites at each data 
collection cycle we should ifce able to increase reviewers' preparedness, 
and decrease consistency artd subjectivity problems to the greatest extent 
possible. Again, the benefits of these activities^should be maximized' 
provided that .our .reviewers 'meet the qua^fic'ations outlined earlier* 
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CpnduCting the Data Collection ' . - ^ Jt^ 

]/fe would like to point out three separate issues which will impact on 
' the^ efficiency and intensiveness "of the data collection process, a^d on ^e.. 
t^ility of data collected. vThese issues pertain to: '/^ ^ 

• .the conduct and ^ordering of the review .process for .individual cases;, . 

• i the sampling' procedures fdjr choosing case5 for review; and — ■ ' 

^ • • the physical location for conducting review activities. 

First, as regards the conduct of the review process we feel that 
the reyiewer should collect all Procedu/al Standards da^ta for any of his 

or her assigned cases. That^is, responsibility should not be split among 
different reviewers, for collecting the R-300'Verification, CRS, and Timeliness 
Assessment data. In our opiition, those three data collection instrunaents 
complement each other in terms of the information they elicit/ and in terms 
of th^^uKderstanding of the circumstances the case which .results when 
only orte review^er collects all the information required*bn th^l^ase. We would* 
thu§ repommend that one reviewer conduct the full data collection process for 
any^ given 'casV, and that tJ]e review process be conducted in the following 

^order-v.' . >^ . ^ " 

1) R-30b Verification ' 

, 2) CRS (or >ICRS) - * ' 

3) Tiifeliness Assessment. 

' As noted earlier, by following this order the reviewer is forced to 
digest as much "objective" background information as possible, an^ should 
have an understanding of the case which is exceeded only, by the counselor 
and possibly by the relevant counselor supervisor. This level of objective 
knowledge and understanding can only help the reviewer in making the 
required "subjective" 'judgements, particularly as rejl^te to Timeliness 
Assessments. 

Another important issue concerns the sampling procedures to be used ^ 
in s,electirig cases., for. review. This issue is related to/ two practical 
concerns; (1) we must ensure that, for each of the Procedural Standards^ 
issues, a sufficient number of cases haVe been reviewed to allow for 
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analysis and interpretation of the data; and (2) we would prefer that, to. 
the e/tent possible, reviews be conducted on clients who are in service 
so as to assess current operations ^nd to allow fpr intervention on behalf 
of current in service cases, ,where problems seem to exist. 

.The first issue arises 'directly out of p\XT experience during the pre- 
test. Specifically, we were unable to ^ss^ss performance on those CRS 
data items 'related to extended evaluation cases, and to cases closed ineligibl 
These problems arose because: (1) the MEU's specifically decided to excludi 
0^'s from their samples; and (2) no provision was' made for ensuring a^ > 
sufficient sample of cases ^yho had |)een placed in ejftended evaluation. 
Closure status was the criterion for -sampling and selection; it was only 
coincidental that some, of the^e cases had gone throu^ extended evaluation. 
These problems can be corrected ' siraplyw by incorporating an additional element 
to the sampling frame: specifically, by including the variable "entei*ed/did 
not enter extended evaluation^' as ond of the sampling criterion, and by^ 




selecting 3 sufficient number o£ cases which did enter. ^ Likewise, fut 

d^aXaT collection -efforts must include OS's, in sufficient numbers to allow 

assesrsm^t of . compliance with the regulations pertaining to ineligibility 

) ' - 

detei;Tninations. ' ^ ' 

Tbe second issue relates to the overall uti;ijity of the Procedural 

Standards," for managing current ageniy operations. Specifically, reviewers 

noted that, since the sample included only clos'ed cases,' -ihere was^ no 

possibility for intervening to correct any problems which mgy^ exist for cases 
♦ y*.^'". * 

currently in service. We feel' that this issue can he, r^d^Vfed fairly simply. 

' * *^ 

''The fact is that each of the data collection instruments and particularly 
the CRS lend themselves to ''segmental auHit^g." That i^V the instruments 
are structured in such a way as to allow assessment of fairly discrete stages 
of the overall service process: for example, the CRS. contains separate 
sections on eligibility determination, extended evaluation^ plan development, 
service provision, and closure. There is no reason why the sampling fifame 

r * 

cannot include cases which currently fall into each of these points of the 
servic^j process). Thus, w^ should target such cases, while still ensuring 
sufficient numbers of closed cases for analysis of closure issues. , 

The' IJI^^ point concerning conduct of the 'da^a .collection relartes to t.he, 
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physical location Qf the <Jata collection effort, Logistically, it is prefer- 
"-•^ble to centralize the data collection activity in one place (e,g., the * 
state's c<?ntral ' offices) However, it was -noted during th^f^retest t)hat 
requiring removal of case fil-es from their ''home base'' (e^g., a district 
office), \/as problematic in cases where the files were .^lee^ed at the home 
base. This problem would be exacerbated if th,e sampling's a^lju^Vd to 
include in-service statttses. We do not have a specific recommended* solution 
to this problem. We wpuyd simply note that the issue should be considered, 
and resolved as approlJriate in the particular state. Pferhaps the reviewers 
could travel to districts.^o conduct reviews, if they pan do it without 
incurring excessive travel or other costs,'' 
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Intet-^^encv Responsibilities for Data Collection and Reporting; and Advance 
Work N?eded Prior to Data Collection' Cycles ~ 

Thus far we have addressed issues of a gen^eral natuVe pertaining to * 
^liscrete aspects of the Procedural Standards system. At this point we 
wish to discuss how these aspects would be brought toget!her, depending on 
where we place responsibility for data collection and repotting. Specific 
♦caLly, we ask: how should the* system work if state' agencies are responsible, 
, and how would it woik if RSA is respojisible?" Ultimately, the decision of where 
to place responsibil\ty must be made by RSA and the states; we clnnot make, 
*that decision/. HoweVerT'Hft what follows, we will present our ^nderstanUing 
of the pros and c^ns assoc^ted with.iiiff erernt ^responsibility structures, 
and will draw^outthe implications of the earlier discussions as they rqlate 
to location of responsibility^ . 

Locating Respons4bility with State Agencies 



IS^Plj] 

5i^i1 



Placing responsibility for data collections and reporting on the 
^various state agencies has some potential -benefits. These include: 

• Reyiewers at the 3tate leveL are closer tS the system under 
revidw. They would have a prior, and oetter understanding of 
state policies, the agency itself , *hnd any peculiarities of the 
N state ^ that miist be taken into account when assessing performance 
qn the Procedural Standards. 
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• The greater proximrty of Estate agency personnel to the system under 
^ ^ review incij^ases the chancesVojr constructive feedback and improve-^ 

ment, in a shorter period of time* 

. But placing responsibility^ at the state level can also have some drawbacks.^ 
These* drawbacks include: * - - \ 

• The possibility*^that ii\any siate agencies will not hav6 the available' 
Manpower to conduct a thorough case review. As ndted earlier, 
given the training and time requirements, we would prefer that the 
review function be made a regular duty of whoever is chosen to con^ 
duct case reviews. However, agencies may ndt be able to support 
such a staff role.- 

• Also; there is the problenj^f partiality (i.e., conflict of \ 
interesr) if responsibiljLty for case review is placed too low 
withinythe state organisation < If the review team is macle up of 
counselors, counselor supervisors, or even dist^rict administrators, 

* * .the review team might end up investigating^ itself: that is, any 
perverse findings might reflect .back to the review team* This , 
could certainly discourage a thorough audit. 

If the first .problem can be resolved^ (i.e. , states c$tn support personnel 

in a* case reviewr role) then the second problem migHt be overcome by placing 

reviewers witliin the evaluation section of the agency administration (or.. 

aitematively, by assigning the review function to current staff in the 

\ ... * 

evaluation division) » As members of the agency's Central administration, * 

' si^ch personnel should be able ^o avoid conflict of interest. 

If states have- responsibility for data collection, w^ strongly 

recommend that the Modified Case Review be used, rather than the full Case 

Review, so as, to minimize effort requiored. Further, for those /states' 

already p.^ssessing a case rey^^w process, we feel it would be , acceptable 

fo-r them to use theii^j^^wn instrument and! process,^ and then transfer' the 

information to the standardized Procedural ' Standards instruments. However, 

the state.' s process must provide^ll the information required for the 

Procedural Standards, and'' the state* must demonstrate to RSA that this is 

the case. To the extent that required information is missing, the state's ' 

process must be modified. 
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The last design. issue concerns the advance\\vork needed to. identify and 

account for state agency idiosyncrasies. ^tVe' f eel that it is the state's 

' * i . - ' 

responsibility to educate RSA about the various* unique aspects of its ♦ 

• » ' ^ - . i- 

operations, so that RSA can have a context for interpreting data results. ^ 
Basically, we feel that RSA and the .^states should together review the data 
requirements ^f the Procedural Standards instruments, and* that the state • * 
should point out where , its uniqye aspects could cause problems for data 
collectionyand reporting. In this process, it will be the duty of the 
state to demonstrate how interpretation should occi% on the problematic 
questions. We suggest that -states dev^Jop a 'Tiandbook** «which -documents 
these problems, so that RSA can interpret thd results and use the results 
in comparing states. This handbook could be updated to reflect changes in 
circumstances, and should be used as a tool during the. pre-dati collection 
training session. In this way, we allow maximum fl,exibility to states, 
while satisfying RSA's information needs, 

Locating Responsibility with RSA 

If RSA is assigned itesponsibility^or data collection and reporting, the 
m 

logical place to put this responsibility is .with the regional office. Within 
RSA, the regional offices will most likely possess the best level of knowledge 
regarding operations within their relevant states. As with state agencies, 
there are pros^ and cons to placing responsibility with RSA regional offices.-^ 
Hie benefits are discussed bel.ow. ' % \ 

• Perhaps the greatest benefit -re*sulting from havinr RSA resjjonsib'le ^ 
is that by doing so-, we position the concern* for inonitoring compliance 
within the RSA structure, .rather than depending on states to assume 
responsibility foi(^this activity. Hiis'is appropriate inasmach *a5 
the Standards address RSA concerns, and since they will be imposed 

by RSA* 

• By placing responsibility^ with RSA regional offices, we should reduce 
the number of people involved in review, and reduce the overall cost 
involvyed in P.rdcedural Standards data collection. f In fact, the 
liature and requirements of the review function lend themselves some-* 
what more to placement within RSA* ra'ther than within states. For 
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example, it makes some sense to place responsibility with RSA, sinae' 
the statewide assessments would occur only every three years,- in a 
-given state. GiVen this/RSA can maintain an on-going review team 
^^^hose responsibilities may or may not include duties other than ^ 
da^a collection for the Procedural Standards)^ m6r,e^easily than 
could states. The team would then rotate among the states' i^T the 
region,- as the 'time arrived for a statewide assessment. This 
arrangement would have the effect of^ ^ > 

- minimizing the training and retraining required; 

- increasing the consistency and reliability of the data base; 

- increasing across-state comparal^ility of data; and ^ ^ 
reducing the cost to the states and to RSA. 

\ An additional benefit jresul ting from RSA^assuming responsibility is 
that, as RSA staffs- tfie revie^vers sholild possess the best understanding 
pf compliance issues possible,' HoN^ver, tor the Procedural Standards 
process to be as useful as possible, the reviewers should ideally 
also possess the other qualities, noted earlier, as desirable for 
personnel conducting the -review. These include: 
^'^an underst^ding of practical casework procedures and ^^est 
pi^ctice"; and ^ 

• - anninderstanding of unique aspects of sta,te operation., 

% 

In other-words, should RSA be assigned responsibility, the people - 
' we chose as reviewers should provide* the 'TDri^ge" needed to be. 
^bl?*^ to- assess state performance vis-a-vis federal concerns, 
while at the same time V^rforming that assessment within the 
conte;ct of the^ staters operations* Thus, the importance of the- * 
»»handbook", discussed above, becomes cl^ar. 

\ In a major sense, placing ^sponsibility with RSA is appropriate 
in that it allows for identif i-cation of Jechnical assistance needs 
by one of the major providers of technical assistance to th-e 
states. This argument is further strengthened by th*e fact that this 
strate^' will ensure the availability of a more complete data base 
for supportive evaluation, particularly if regidnal offices continue 
to use. the full CRS for case reviews. Even without using the .full 
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CRS, RSA^staff could use a. combination of tl\e Modified CRS (so 

as to assjfss performance on the> Standards) with Jhe supportive | 

evafuatipn data items** from the CRS. 

0 Presumably, by having RSA staff conduct the data collection^ 

less partisanism will be ^hown on the. part of'data collectors, resulting 
in decreased scepticism of the data's validity. , * 

• Finally, by having RSA assume responsibility, we should improve 

the overall receptivity of the states to the full standards system 
* *** # 

(i.e.. Performance and Procedural Standards) . 

In terms pf drawbacks, we see two major ccJbcerns. 

• First^, RSA reviewers might not be able to become familiar, enough 
Vwith the peculiarities of the various states under review, on a \ 

continuing basis. If this proves true; the findings of reviewers 
mi^t l^cgj'distorted or invalid. However, we feel that if our 
suggestion's regarding reviewer qualifications, experience and 
training (e.^g. the ''handl?ook*') are followed, this should not 
prove to be a major problem. ^ ' - , / 

• Second i» due to the bureaucratic *'distance*' between the states ^nd 
RSA, we may experience unacceptable time lags between data collection, 
reporting, provision of technical assistance. Howevfer, any such 
'problems would presumably be reduced as a result of working the 

'iDugs'' out of the system, after implementation. 
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' X, CONCLUSION 

The Vocational Rehabilitation Program Standards have now undergone 
three iterations of development. In an earlier chapter of this repor/t, 
we discussed the process by which the current RSA VR Program Standards 
were developed, and we discussed the later developmental efforts which 
culminated in Berkeley Planning Associates' recommendations for a revised 
set VR Program Standards. This revised set of standards has now under 
gone a field pretest 'in six ModejJ Evaluation Unit (MEU) state agencies, 
resulting TLn a final set of recommended standards, data elements and 
data collection procedures. This report presented detailed discussions 
on each of the standards „ data elements, and data collection procedures, 
with respect to the following issues: ^ 

• the rationale for including the standard or data 
element in the overall evaluation systemj • ^. 

• the process for training and data collection required 
to conduct th^ pretest; 

• the time investment required of MEUs to conduct the ^' 
> pretest; . • ' * 

• the analysi/^^ults for each data element, using the 
pretest^data; 

• the comments and reactions to the standards, data element^ 
and data collection forms, obtained from RSA. reviewe 
the NEUs, and other state agency reviewers; and 

• our responses to those comments./ 

Throughout the pretest, BPA has attempted t^ weigh the. pros and cons 
-^relevant to each of the standards, data elements, and data collection 
instruments, in ^terms of their utility in furthering RSA's broad, evalu- ' 
ation and program management aim^, their .methodological and conceptual 
soundness, and their feasibility for implementation. But of this process 
has come our final recommendations for the various c^nppnents (i.e., 
standards, data elements, and data collection procedures}^ which will com- 
prise the total VR Program Standards System. These recommendations are 
Siammarized below^^ ' ' * ~ 
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Recommendation's for Pufcram Pcrtormance Standa^rds 



S 



S/andards 



Recommendations 



VR shall serve the maximum proportion of the potentially eligible targat^ population, 
subject to the level of federal program ftJnding and priorities among qfients. 

The VR program "shal 1 use resources in a cost-effective planner and showla positive 
return to society of investment in vocational rehabilitation of di sabldjd Jcl ieTits» 

VR shall max imlii. (he "number and proportion of clients accepted for s^pjices who 
are successfully rehabilitated, subject to the meeting of other standard^. 

Rehabilitated clients sjiall evi/dence increased econbmjc indQ'i>endencc. ^ 

I ^ > 

There shall be maximum placement of rehabilitated clients in^o^ompetitive employment. 
Non~compct itive closures shall be in accordance with the IWRP goal and shall represent 
an improvement in gamfiil activity for the client. , 



Retain as is* , 
^ 

Retain as is. 

Retain as is. s 



6. (Old) Vocational gains shall be attributa^^le to VR services. 



(.New) Rehabil JtatX'd cUents shall evidence vocational gains. ' 

7. Rehabilitated clients shall retain the bertefits of VR services. 

I 

8. Clients shall be 'satisfied with the VR program, and rejiab'i 1 itated clients sha.ll 
appraise VR services as useful in achieving and maintaining their vocational 
objectives. • • 



Modify: delete the word ••'increased," 

Modify: delete "...shall be in acco^r^fl- 
* ' ance with the IV^P^ goal and..." 



Eliminate fi^om Performance Standards. 
Establish causality through periodic 
controlled research studies. 

New standard. 

Retain as is. 

Retain as is. 
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] As well, this report presented a summiry^ and a working example of 
'our proposed^odel for using standards information to identify problems 
in state agency perforitance . With* such a system for using program data 
to support program management and decisionmaking, 'the standards will / 
more effectively serve tJheir original function: to assist RSA and sxate 
agencies in directing the VR program towar'ds 4mprQj/j^d performance. 

SUMMARY OF RECOiMMENDATIONS FOR THE PERFORMANCE STANDARDS^ ^ 

Using the pretest data results, and aftfer considering the" comments 
we received from the MEUs (both before and during the pretest) and from 
other reviewers, we have developed a revised set of recommendjed Program 
Performance Standards and data element;s. Qur primary concerns have been 
that any standard or data element recommended for* inclusion in the final 
set meet the following kinds of criteria: 

• Standards must address, and data elements must jfneasure, 
achievement of a valid VR' program goal, . 

• fhe VR program must be, ca pable of influencing performance 

♦-^ - ' ' 

^ toward the goals addressed/measured by the standards and 

datar elements . 

Any data element recommended for inclusion should be non- 
redundant with other data elements; should allow' for assessment 
of a giveti state's performance independently of other states ' 
^ performances (i.e., no "closed systems"); and should have a 
strai^tttforward interpretation in terms of 'S^ood" and "bad" 
performance. 

The recommended sets of revised standards and data .elements appear 
in Tables 55 and 56. As shown in Table 55, we recommend that five of 
the eight originally-proposed Performance Standards be retained in their 
current forjn. Of the other three standards, two should be modified 
slightly. We suggest that Standard 4 be modified by deleting the .work 
"increased," inasmuch as we feel VR's responsibility is merely to facili- 
tate client achievement of a minimum acce^^table level of economic inde-' 
pendence. l^ile we applaud any program success in facilitating clients' 
achievement of even greater levels of economic independence, it would be 
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unreasonable to require more than the minimum income needed for an acceptable 

level of economic-independence. 
,/^ti.Standard 5 shoiplA blfeodi-fied by deleting the phrase "sJiall»be in 
^ ac^Fdance with tH'4 *J^Bngoa:l.. " * This modification should remove any 
« incentive to ^^settle" 'for "©onrCompetitive outcQ5stfs]\even if a non-competitive 
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goal was seen, at the;j|ilt?ij:3^v^^ thafeest a particul 

client could hope for. - \ - 1 Jt 

We recommend that the ^'ceuasality" standard (StaTOard 6) be eliminated 

from the Performance Standards. While it is important from an evaluation 

standpoint to estabtish a causal relationship between VR services and ob- 

, served benefits,, a valid |^ethodology for dping so has yet to be developed.. 

. We recommend that causality be establisjied through controlled research 

studies, occuiyring less frequently (e.g., every third year) than the basic 

IP 

annual standards^eport ." 

' Finally, in the place of the causality standard, we recommend a new 
Standard 6, mandating that '^rehabilitated shall evidence vocational gains.** 
Thus we retain a concern for documenting positive client change, while 
removing "the requirement' to "prove" VR's role as the primary change-afgent . 

With thes^ modifications, we feel that the revised system of Performance^ 
Standards captures the broad Qoncerns for evaluatdng^he VR program's* 
achievement of coverage,, impact, ari^ief f iciency. The same kind of thinking 
process occurred -in considering which of the data elements --.the discrete 
measures of goal achiev^ent -- should remain in the re'\{ised standards 
system. As Table 56 shows,* we recomiiiend that 11 of the original data ' . 

elements be retained^ in their current form. We feel that -four of the 
data elements (or component^ of data elements) should be e^iin\inated com- 
' pletely.from the standards, for the following types of reasons^ 

The data element represents a closed system. This is the ca^e 
for data element l(i). , ^ 

The data element implies^ goal which is eithef too stringent ^ 
(i.e.., goes beyond what is reasonable to expect from VR), or ' ^ 
which is inappropriate as a performance indicator. This is 
the case for data elements 4(iv), and for the "satisfaction . 
with present situation" item 'in 8(ii). _ - 

Ttie dat^ .element rests on unacceptable methodology, as with 
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' Revised Recommendations for Program Perforroancp Standards Data Elements, August 19, 1981 



Data Elements 



Recommendations 



Standard 1:* Coverage 

(i) Comparison of caseload served to expenditures 

(ii) Clients served per 100,000 population 

(iii) Percent severely disabled 

Standard 2; Cost-Effectiveness and Benefit-Cost 



(i) Expenditure ^per competitively employed closure 

(ii) Expenditure per 26 closure » 

(iii) Ratio, of total VR benefits to total VR costs (benefit cost ratio) 

(iv) Total net benefit from VR services (discounted net present value) 



Standard 3: Rehabilitation Rate 



(i) Percent 26 closures 

(ii) Annual change in numbex; of 26 closqres 

Standai^d 4: Economic Independence 

(i) Percent 26 closures with weekly earnings at/above federal minimum wage 

til) Comparison of earnings of competitively, employed 26 closures to earnings of 
employees in state 

(iii) Comparison of earnings before and after VR services , 

(iv) Comparison of 26 closures with public assistance as primary source of support 
before and after VR services 

Standard S; Competitive Employment * 
(1) Perc<y>X 26 closures competitively employed 

(ii) * Percent 26 closures with hourly earnings at/above^ federal minimum wage 

(iii) Percent 26 'closures with competitive outcome or with non-competitive outcome 
, and* non-compet iCive goal 

(iv) Percent non-competitively employed 26 closures showing improvement in gain- 
ful activiity 

Standard 6; Causality (original Standard 6) 

(i) Comparison of earnings change from referral to closure of 26 closures to- 
earnings change of a control grSup 




Eliminate from Standard. Retain as inform- 
ation item in reporting system 

Retain as is. 

New data element. 

'Retain as is. * 



Retain as is. 
Retain as is. 
Retain as is. 

Retain as is.* 
New data element. 

Retain as is* , 
Retain as is. 



(/4 

00 



Modify and move l.o Standard 6 
Eliminat.e 



Retain as is. 

'Add "competitively employedl" 
Move to Procedural Standard! 

Modify definition of "non-vocational 
improvement," using client-oijienteJ F*A.I. 
and L.S.l. measures. 

Eliminate 
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|g Tabic 56 (cont.) ^ ' 



Data Elencnts 


1 « 

Recomnendat i ons 


StanJard 6: Client Chnnjce {oev» Standard 6) 




(i) Average [jre^post VR >crvicc earnings cl^inge for 26 closures 


New data element. Modification of 
4(iii). 


(ii)* In addition, changeij in statuses and functional ability, when such measures 
become available < 


Data (dements to be developed. 


^trmd'irf! 7* fttitontion of Rr»nr*fit'^ ' 




(i) Percent 26 closures retaining earnings at follow-up 


Retain as is until SSA data links 
- established annually. 


(ii) Coiiiparison of 26 closures with public assistance as primary source of support 
at closure and at follow-up * 


Collect income estimates from all sources 
of support, and ascertain if >S1% is PA. 
Done at closure and follow-up V 


(iii) Percent non-corapetitively employed 26 closures retaining closure skills at 
• follow-up ^ ; ^ ' 


Modify definition of "non-vocatibnal 
improvement" using 'the same client-oriented' 
F.A.I, and L.S*^I. measur^ as in S(iv). 


Standard 8: Client Satisfaction and Utility Appraisal i 


(i) Percent closed cUents satisfied with overall VR experience 


Retain as is. 


(ii) Percent closed clients satisfied withi present situation 

information provided 
counselor, pron^tncss 
' ^ nhvsical Tf»^l'OTfll'i nn 


- Eliminate "satisfied -with, present 
situation." 

" irfumuine counsciur bai.A^bAaci.iun luciuo. 

- Retain service satisfaction items, 

^ : ■ C 


*job training stjrvices ; ^ 




placement services 




(iii) Percent 26 closures judging services received to have been useful i?i obtaining 


Combine with 8(iv) in new 8(iii)* 
Combine with 8(iii!) in new 8(iii) . 


(iv) Percent 26 closures judging services received to be uspful in current performance 
on the job/homemaker situation 


411 .. , 
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. In one case for data element l(i) — we suggest that the data be 
available* in the reporting system for RSA's use7^ut''*not used for pur- 
poses of evaluatitig state agency performance. ^ / 

In addition to the data elements completely eliminatad-^om the 
Standards system, we suggest that data element 5(iii) be moved from the 
Performance to>the Procedural Standards. .This is in, keeping with the 
•modification >f the wording of Standard 5. The program should maintain 
its concern for good planning for services and outcomes, but such a con- 
cern is not 'app2:opriate as a perform^ce evaluation measure. Also, we 
wish to^ avoicf any incentives to 'settle for non-competitive outcomes 
in those cases wherever ceive.d potential "goes up" (i.e., points to com- 
petitive employment) during the service process. Recai3ting the issue as 
a [Procedural Standard should resolve these proBlems. 

* We suggest that modifications 'be made to eight of the data elements ♦ 
We recommehd modifying the form of data element 4(iii) and moving it to 
the new Standard 6, measuring client gains. The original data element , 
measures the total percentage change in client earnings, pre- versus post- 
VR. In its modified foi:m, the data element measures the average raw 
dollar change in Mient earnings: 

We recommend a change in the computation of data element 5(ii), to 
include only competitively-employed rehabilitants in the calculation. 
This focuses. th^ data, element more clearly as a measure of gainful employ- 
ment — *and quality of employm^nt — among competitively-employed closures. 

• Two of the modified data elements -- 5(ii> and 7(iii) — pertain 
specifically to, non-competitively-employed rehabilitants I In both cases, 
we recommend deferring data collection until after*, the conclusion of the 
upcoming pretest of the ^nctional Assessment Inventory (FAp/Life Status 
Indicators' (LSI).'' This pretest will determine the FAI and L'SI measures 
to be included in the MIS design, and we recommend that the standards use 
the same measures to assess gains for non-competitively*-emplqyed closures 
as are chosen for the MIS. This strengthens the link between the standards 
kn^ the. MIS, and avoids proliferation and 'overlapping of data elements. , 
To implement data element 5(iv), the FAI/LSI measures must be included 
on the R-300 (in both the section completed at acceptance and the ^^ection 
completed at closure). To implement 7(iii), the FAI/LSI items, used to 
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produce the relevant, measures/ must be reformatted to allow for self- 
administration via the Follow-up Survey. 

The remaining mo^dified data elements pertain to all 26 closures. 
We suggest that the definition of **primary source of. support" in data 
element 7(ii) be changed such that (1) t'he>^eaning of "primary", becomes 
intuitively clearer, and (2) that the focus becomes one of public versus 
private sources of support. The remaining modifications all concern the 
satisfaction issue (Standard 8) . ' The service satisfaction items in, data 
element 8(ii) should remain separate. However, analysis o£ the counselor 
satisfaction items (data element 8(ii)) and^service utility items (8(iii) 
and (iv)) suggest that these two issues could be ^adequately addressed by 
combining the original five questions into one question on satisfaction 
with counselor and one question on service utility. These suggestions 
.ar^ reflected in our revised questionnaires-. 

In addition, we recommend adding two -new data .elements modji»fying 
various pata elements, for which data are immediately available. ^In both 
cases, tne additions are recommended so as to more completely reflect the 
goals implied by the wording of the standards. Thus, for' e)cample, a 
"percent severe" statistic should be included as a .data element in Stan- 
dard 1, to reflect the phrase "subject to . . . priorities 'among clients." 
As well, we suggest adding a data -element to Standar.d S.^which measures 
annual chang.e in the number of 26 closures,- to reflect 'the phrase "raaximi 

the ... number ... of clients ..."rehabilitated." 

* — T ' n . 

'Finally, we recommend including a new data element -in the qlifent l 

change standard, addressing the gains of non-competitive ly-employed 

rehabilitants. Like data elements Sfii) and 7(iii)> this data element 

is contingent* on further development of functional and status measures. 

The recommendations detailed above are reflected in our, recommended 

changes to the data collection instrum^ts. .The R*300 will require some 

additional information. However, it will, by the same token, become the; 

primary source of client-level data for the Performance Standards. The 

Closure Survey will be used solely to address the satisfaction standard, 

and has been reduced from 26 to 6 (questions,' totaj. This should gteatly 

reduce the costs of administering the Closure Survey, and should increase 

overall response rate. Likewise, the Follow^-up Survey has been reduced 
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from 26 to 6 questions^, and will be used solely to address issues regarding 
retention of benefits. * . 

The changes described above round out our recommended revisions to * 
the Performance Standards and data elements, and to their corresponding 
d^ta collection' instlments. We feel that, with these revisions, the 
standards comprise a set of goal statements and evaluation indicators ^ 
which is comprehensive, appropriate to VR goals, and conceptuafly and 
methodologically sound. 

SWfl^ARY OF RECOMMENDATIONS FOR THE PROCEDURAL STANDARDS 

We feel that the Procedural Standards and their associated data, 
elements adequately and accurately address RSA's concerns for 'data quality, 
service timeliness, and compliance with the legislation. As such, we 
•recommend that the' Procedural Standards and their associated data elements 
all be retained in the revised VR Program Standards system. In addition, 
as dis<:ussed earlier, we recommend moving the originally-proposed Performance 
Standards dat^ element SCiii) (goal-occupation match) to the Procedural 

Standards. ' ^"^^^ ^ 

The major issues with the Procedural Standards. corjcjKTi the process 
])y which the standards are implemented. Table 57 summa^zes those recom- 
me;idations.* We recommend that Procedural Standards^data collection occur 
in a given state only once every three years. \ 

We have recommended a wide range of qualifications to look for when 
selecting reviewers, and^we have noted threes specific 'issues with regard 
to data collection (e.g., orderiii;g of review, sampling, and training o£ 
^reviewers). Finally, in the previous chapter, we presented the pros and, 
cons associated with placement of responsibility fOr data collection \fith 
the states, versus with RSA. W^ feel that, with careful consideratipn of 
these issues by RSA and the states, and* subsequent implementation based 
on this consideration, the. Procedural Standards will* have their desired 
impact on tjie VR system in terms of 

• encoura'ging compliance with the legislation; and 

• ' improving the quality of VR services and their cost- 

effectiveness." 
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Table 57 . 
J^ecommendations Vor tfie Procedural Standards 



1. ' Reta_^in the standards on R-3)D0 validity/ eligibility decisions, 

IWRP provision^ and service timeliness, 

2. Retain, all data elements relevant to the above standards. 

3. Add a new standard on goal planning (former data element 5(iii)). 

4. Conduct -statewide proceclurai standards data collection every three 
years, _ _ - . - • - . 

5. Retain the data collection instruments as pretested, with the 
^ following qualifications: 

if states conduct the data collection, have them use the 
Modified Case Review (a shorter version of the basic Case 
, Review Schedule); 

--•add an item" to the Timeliness Assessment to identifyAcases 
handled with **undue speed." V 

conduct special validrty checks of key R-"300 "items , if desired, 
in the same year as the Procedural Standards data collection 
activity; ^ ^ 

6. Seek reviewers with the following qualifications: 

— >familiai*ity with unique features of state operations; 

— '^experience in case work and administration; 

4- no conpict pf interest. . _ . 

7. Attempt t^ maintain continuity of" reviewer personnel. 

8. > Conduct a comprehensive reliability assurance procedure at the 

start of eagh^ new^dala,, collection effort in a state agency. 

9. Assign the -full P^cocedural Stahdar3s da^'collection, for any . 

_ given case, to only one""reviewer , to -ensure familiarity with the 
circumstances of the cas^^ 

10. Conduct the 'review in the following order; 
.^.^-iv- R-300 Validity,__ ---^^v_^->_ ^ ' - 

2^ Ca^e ^Review Schedule 
3. Timeliness Assessment ^ 

11. Sample a sufficient number of 06 participants and 08 closures, so" 
as to allow analysfs of the Procedural Standards issues relevant 

' to th'(5Se groups. 

12. Sample a sufficient number of in-service clients to allow analysis 
of current case. practices. ' . 
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APPENDICES ' 

1. Identified Agency Concerns from the R-300 Validity 
Pretest 

2 . Case Review Schedule Developed by San Diego State 
University T^CER IX . 
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APPENDIX 1 

IDEiNTIFIED AGENCY (/oNCERNS FR0!^1 THE R-500 VALIDITY PRETEST 

The R-300 Verification procedure revealed to the agencies a wide 
variety of concerns with their case record systems. Many of these prob- 
lems could be categorized as inaccurate, inconsistent or inadequate re- 
cording of case information. The remainder were unique problems arising 
in individual states. These cpmments^ were taken from the Participant 
Evaluation and CommeiH: Forms #2. 



DELAWARE Differences. between districts. 

Lack of compliance in some areas of case documentation. 

\ 

Need to streamline case recording. 

Basic skills needed by counselors and other classes of 
service delivery personnel. 

Disability codes not necessarily placed in correct order 
(primary vs. secondary) on IWRP and Rr-300 data (DVR 313). 

MICHIGAN As long as some validating was being done, I felt we should 

do validation of entire' R-300 — if it would be of any use. 
And, if it is of no use, why are rec.9?'4ii»g it in its 
entirety? -..r*>;' ^ ; ; 

Need for updating of recorded iiL^sfms^ii^^ circumstances 
Grange. ' Uv^'-' ' 

I feel we should be checking the disabilities closer and 
the severities before submitting the -300 documents. 

Things are done in k careless manner. Workers have the 
feeling no one .checks; all they want is for all the blanks, 
to be filled in. I am much more aware of the need for 
casework documentation, rather than picking the informa- 
tion from- material not in the case file. 

The number of questions posed for clarification purposes 
by these reviewers (even those working with those* forms 
on a daily basis). Does, this confusion exist district- 
wide? Any way for improving the R-300, series so that it 
V . is more clear? " , • 

Various items on the ,forms are open^^to different .inter- 
pretation. 

I don^t sel^ any concerns about the 300 system. The few 
• mistakes could have been overlooked. As far as reporting 
.data, I feel the 300 system is sufficient. 
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Deteraing the specific problems that h^pen among coiv- 
selors as relating to, casework and documentation; recom- 
mendations to improve the quality of casework. 

Poor IWRP; inadequate case^ecording; clients not advised 
of confidentiality of info; no certificate \f ineligibility 
for statuses- 28 and 30. ' * 

Lack of order in case folder filing. 

Failure to answer "all" questions on R-300 anU R4. 

Inconsistent coding <>f benefits (SSDI, SSI, PA^ • 

Unsubstantiated coding of severe dis^ility^. 

Obvious failures to pursue doctors^ reW)mmendatiV)nj . 

Apparent misapplication of eligibility cHteria 
mentally ill cases. ' 

Inappropriate eligibility determination. 
Inappropriate SD detemiiriatioiL *. 
Inadequate narratives. 

Inconsistent filing/record keeping procedures. 
SD coding inaccuracy. ' i ' 

Failure of counselors to follow through with "thoroi/gh 
diagnostic assessment" and/or' to follow recommendations 
of medical consultants. 

The acceptance of cases who are borderline eligible or 
even clearly not eligible. 

Medical consultation sometimes sloppily recorded omit 
to indicate whether severe or not. 

Documentation of R-114 medical consultant judgments some- 
times skimpy or even seems to contradict consultant ^s 
judgment^ ^ - \ . 

Sometimes seems' to be little evidence of direct,,^CGunselor- 
client contact. 

Sometimes inadequate follow-up to find why client is not ' 
responding before closing unrehabilitated. 

Uniformity of judgments. . 

Validation of computer runs (e.g., months in status). 

Consistencey of coding of SD and major disabling conditions. 

Need for simplicity of verification for SSI and SSDI pro- 
grams . • ' 

Need to standardize procedures. 
Need to lighten burden on counselors. 
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Agency* maybe penalized or negatively rated l)ecause certain 
aper;vork aspects are not employed^ in the case proqess and* 
t the info may be inferred or interpreted from other case 
data. In our use, the info ^'makes sense" but may not be 
considered "uniform'* for^a third party." ^ 

I- think .the ar^as of concern were in Trust Fund, Supple- 
. ^mental Security Tund and SD check-offs, because many 
questions were brought up about •these during the review. 

Correct posting of funds expended; better training in com-^. 
plating a 104 more thoroughly; more spe.cifi.city in declar*- 
T ing severely disabled and rationale^ 

Accuracy of tTi^ input documentation. 

Indicates that some problems exist ,iii respect to inter^ 
p^retation of some aspects. of casework.^ ^ 

Uniformity. 

Too cumbersome. ' ^ 

Collection of useless data or data not directly connected 
to proving eligibility and the provision of VR-.services 
to achieve SGA for the client. 

Most of the discrepancies noted were minor and did not 
affect federal cost participation in Special Programs. 

'Concern about counselo;r judgment, / 

Concern about training. ^' ' . 

Concern about who is extracting information for repo^;;^T||^, 
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APPENDIX 2 



CASE REVIEW SCHEDULE DEVELOPED BY SAN DIEGO STATE UNIVERSITY RCEP IX 



JThe following data items are used for the Procedural Standards: 
Standard 10: Eligibility 

Section II.A*: 4-11 . ^ ' 

Section II. B.: 14, 19^ 33, 40, 42a, 42d \ 
Section III; Ic, 3, 7, 8 

Section Vii: 2, 5, ea, 6c, 7, 8, 9, lOa, lOd, 11, 12, 13a, 13c, 14, 17a, 



17c, 18, 20, 21, 23, 24, 25, 28a, 286, 34, 35, 36, 37a, 37b, 



•38, 39 



Standard 12: IWRP ^ ^ 

Section II. B,: 18, 19, 20a, 20b, 21a, 21b, 21d, 21e, 22, 23, 26, 28-^9, 
42a, 42b, 42c, 42d, 43-54B, 43, 54C ^ ■ ) 

J Section V: l,-2, 3a, 3b, 4a, 4b, 5, 6, 9, 11-23, 24a, 24b, 24c, 24d, 
25a , 26 
Section VI: 1-14B, 1-14C 
Section VII: 20, 22, 34^ 
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Section I: 



SectiQ/1 II; 




Section <gfl: 
, Section IV: 

Section V: 
Section VI: 
- Section VII: . 
Section VIII:' 



CASE REVIEW SCHEDULE 

Table of Contents ^ * • , . 

Case Information ^ . T " 

A. ' Cdse-.Iderftlfication. 1 '\ : ; 

B. Sigai/icant Data V la ?• /i 

C. R-300 Verification • • • v * • • — • Ic 

Evaluation of Rehabilitation Potential ^ j * 

'A. Preliminary Diagnostic Study ^- P 

B. Extended Evaluation : * • • 6 ^^^^ 

Eligibility . . . , C ^ •IS- /\ / 

Evaluation of Rehabilitation Potential , ^ 
Thorough Dtagnostic Study - 15 ' 

Individua;med Written Rehabilitation Program 17 

Delivery of Services ' ♦ • ^'^^^1 - 

Termination of Cases 23 \ t ' 

Timf»liness Research and Comments • 7 ^ » * 

A. Timeliness • • • *33 j;?.-^- 

B. Reviewer Comments ' 54 
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1?<STRUCT}0K$ FOR SSCTIWS l* mi . 

^^^^^ ♦ 

To complete the questions in Section i through yiJI, please follow- tnese specicl instroctions-: 

1. Redd the directions at the top of ^ea A pace and within each page, when ^)rovided. 

2. Answer every question unless othervWe instructed {see Kanual; Case Review Schedule Instruc- 
tions:) . . 

3. P»ark the appropriate answer space to the right of the question. 

4. Do not fill in more.than one space for any single row. That is, give only one response to 
each 9uestion. 

5. use a soft lead pencil (»o. 2 is ideal) and observe these irportant requirements: 

Mck'e heavy black marks that fill the circle. 
Erase completely any answer you wish to'chance. 
Kake no strav fnarkincs of any kind. 

6. Tnis scnedj^e is cesicr.ed for r.cChine scoring of your responses . Questions are answered by 
rr,c^K:r.c the cpp»"cpr\cte soswer spaces as illustrated in tms exar^le: 



c 



CuistioK - Wnich is the oniy marking ins-trument O ^ © 

that will be nead properly? , ' ^ , 

Correct Answer 
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S£CTJ0:M: XASt 'IKF^WMATIOK 

A. CAS£^ID5tn'lFjCAT30K 



.ANSWER ^ 
(Write in Space Provijdec-)' 



I 



1 lor individocU who were previously clients, 
I review only frorr* date cese nost recently 
opened. 

"Answer ell iterss usino code nunibers in 
} scu2re(s) provided, ^ef^r to code sheets 
in.'ir^ffoc'i for iter^ noted with (code) end 
■ errter appropriate code on this page. 



I 



I 
I 

1 
I 
I 
I 
I 
I 
I 
I 
I 

f »■ ■ 

ERIC 



■y 



1. Reviewer Initio 

2. Reviewer: 



(I) States (2) Fe^Jercl; 
(3) Other (specify)^ 



3. Reviewer: ^ 

tl) Administrator; 

(2) Supervisor; 

(3) Counselor; 
{A)>Other (specify )_ 

4. Date of Review; 

5. State Aoency (cooe) 

6. Case Nusiber: 

7. Case Status: 
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SECTION It 

* B: 



CASE ir^FOltHATION 
SI<R4ir|CAirr DATA 



Doe? the client's case reord contain a copy 
of the case service report R-300 or the 
equivalent state statistical reporting form? 



NOTE: I^1?0RTA^^X IN'STRUCTIONS 

Respond to all applicable 'items 9-19 utfing in- 
fomation gathered fom the case record • 

In the event the inform^ion is not found in the 
case record, use available data from the R-300- 
or its equivalent. If the data is not recorded 
(VR) aaywhere in the case file, fill in circle 
5 in the far right; cola-nn. If the data* is not 
applicable (U/A) to the client, fill in circle 6 
m the far right caijunn. 

If the case Acord contains an R-'300 or* equiva- 
lent fora ^ill in. a number following question^ 9 
and 16, m the column labeled "^R-BOO Verification" 
or the far right column indicating one of the 
following potential findings-: 




^ Verified: 



2 ' Mot Verified; 



R-300 data included and consistent vith that 
found in case record ' < 

R-300 data different from that found in case 
record ^ 



3 = Case Record Only; Data recorded in case record only, ^ot on R-300 



4 ^ R-300 Only; ^ Data recorded on R-300 only, not in case record 

* 5 »'Not Recorded ,(N'/R) ; j^fJata^ 



not recorded in either case record or R-300 



6 3 Not Applicable (N/A) ; Data not applicable to client 



A!(SW£R 
YES NO 



TT 9« Date <?lient was referred to State VR ij^ency: 



10. Signed application date: 



11. Date certified as accepted for 
extended evaluation: 



12. Date tWRP written and approved for 
extended evaluation: 



m ED ED - 

. YR. MO. DAY 

m m m 

YR. • MO. DAY 

L 

m m ED 

YR. MO . DAY 



ED ED 



YR. MO. DAY 



R-300 
Verification 
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SECTION I: CASE INFORMATION 
B. SIGNIFICANT DA'h 



i 



13, Date certified as accepted for vocational* 
rehabilitation services: . 



14. Dat^ initial IWRP written and 
approved; ^ 



15. Date of initiation of VR services: 



16. Closure Date: 



17. Date IWRf^ amended and approved for post 
erap^oyment services:*^ 



18. Date, of initiation of post employment 
services: 



19. Date of termination of^post eirployinent 
services: 



Cn m m 

YR. , H0\ DM 



m ED EH 

YR, MO- DAY 



ED m ED 

YU. MO. DAY 



EEI EEI ED 

. YR. MO. DAY 



n m ED 



YR. • MO. DAY 



ETI EEI EEI 

YR. Mp. DAY 



CD ED ETI 

yrT mo. day 



NOTE: If«PORTANT INSTRUCTIONS 



If the client's case file does not contain 
an R-300 or an ^uivalent form, proceed 
to sectioit II, pw4 to complete the 
section on evaluation of rehabilitation 
pdtential. j ' 

tf the ca|^e file doers contain an R-300 or 
'its equivalent please respond to ail ite^s 
in Section I:C* ' * 



R-300 
Verification 
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-f^'j-^ZOii 2: CASS INFORMATICS 

C. H-300 VSJ^-JCATIOK 



AKSK5K COX-IMC 



^ KOTS: IM?dR7«rr INSTRUCTIONS 

respond to the iaforciation itcns listed 
^iow (20-58> by filling in the nucber Jin 
the «swer coluah which indicetcs your 
bindings reletive -to the .aerccmcnt of 
data found or. the'R-300 (or cctivalent) 
to that found in the Case Ritcord. The 
following is, a delineation of potential 
• responses aj^q their definitions: 



1 « verifi^; 



2 » Not' Verified; 



H-300 data included and consistent with that found 
in the case record ^ 

R-300 jcata different froa that found' in the case 
record 



2 » Case Record Only; Data recorded in case record only, net on R-3O0 
4 « R-200 Only; Data recorded on R-300 only, not in case record 



5 a Not Recorded; 

(N/R) 

6 * Not Applicable: 

(K/A)^' J 



Data not4recordec xn either cSse record or R-300 



Data riot applicable to client 



20. ^ The client's Social Security nun:ber 



21. The client's SSDI status at referral 



22, The cli-ent's SSI status at referral 



23.' •The Rajor (pr ijnary) disabling condition 



24. The client's secondary** disability 



25* TheVc4.ient*s worX status at referral 



26- Tne cllent*s earnines the week prior to referral 



/ 

oi public assist 



27. The clients receipt or non-receipt public assistance at 
referral (Gocuaentation cf hon-reccipt would occur when the 
P-300 records nc public assistance received and the case record 
provides no docustent^tion of receipt.) 
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c. R-i300 vzKiriCAr:o*v 



The 'C\'pe(s) cf ?-jJblic assistance received at refcftal 
a. SSDZ 



b. SSI-aced* 

c. SSI-i:ainc 



-cis&biec 



f. Other ( specif >•)_ 



29. The ncr.z:-ly arvour.z'cf public assistaLOce received at referral 



The ier.cth cf tine, pricr to referral , during which t^e client 
received ouJblic assistance . - 



The arcropria^teness^of the Federal Special Program identification 
checks * * 



a* Social Security Trust Fiinds (TT) i 

b. Veterans (\™) 

c. y.icratcry Acricultural Wo'rkers (.»1AW} 

d. Offender {?0) 

•J 

e. work Incentive Procrar. {Wa^) 

f. Expansion Grant rroject f&C4)^ 
:urity Incdafre Funds (SF). 



^c. Supolenent&l Se 



h. Severely Disab:ied (SD) . 



2- Tne c:.ier.t'£ SSr: stetus at ojLOSJre 



Tr.c c!.ier.t''£ £SZ 5tctus ^t. closure 




-Id- 



c 

o . 



5 "C* ^ 



c 



C O 



a 



c 
C 



o 
c 



n < 

C& 2 Z 



© © 0 © © 0 
©©'©©©© 
©©©©©© 
©©©©©©' 
©©©©©© 
•©©©©©.© 

©©©©©'© 
©©©©©■© 



©©©©©© 
© © © © ©'© 
© © © ©'.(J) © 
'(i)©©©©© 
©•©©©©© 
©©(4)©^©© 
©©■©©(&©• 
©©©©©© 



© d)© © © 



1 



SECTIO?i I: CASE IsrOR>!ATlC:; 
7-' C. R-300 VrRiriCATIW 



ANSWER COLUMN '* 



34. The client's. work status at closure 
'35. The client ?s weekly earnings at closure 

36. The client's receipt or non-receifn: of public assistance at 

closure (Documentation non-receipt would occur when the R-300 
records no puttie assistance received and the case record provides 
■ no documentation of receipts) • ^ ' 



The 


typejs) publ 


^» 


f 

SSDI 


b. 




c. 


SSI-ijlind 


d. 


SSI-disabl«d 


e. 


AFDC ^ 


f . 


Other (specify 



38. 
'39. 
40. 

41. 
42. 
43. 
44. 

45. 

46. 



The monthly anount of puolic assistance received at' closure 
The client *>s oc\:iipation at closure 

The correct. DOT code for that occupation? (Check in Dictionary 
of Occupational Titles) * ^ 

The client's outcome status * 

"Reason for nonrehabilitated closure 

The total cost of all case services 

provided in 



The total cost of all case seJtrij 
rehabilitation facilities 




The total cost of cise services charged to social security 
trust funds * . » 

The total cost of case services charged to supplemental 
security income funds . 
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.TICK 2: CASE IKFORMATION 
♦ 

R-300 VSRiriCATION- 



>*STSUCTIONS : Pleftse answer the 
ollovinc questions (A &nd £) fo: 
e«ch of the- service catecories 
^7-52 listed at ^3je rig^rt. 



ITEM 



SERVICE 



47 



Indicate your finding recardinc the 

agreenent of data found on the R-300, 

or its ecuivalcrtt, to that found in 

the case record relative .to the . ' 

client's receipt or non-receipt of 

the following services. Verified 

• Not Verified 



Indicate yp'ur findings regarding tjie 
agreement of data found on the R-3 
or its equivalent, to that found in 
the case record relative to the cost 
status of the following services. 




Verified 
Not Verified 
Case Record Only 
, 300 Only 
N/R 
N/A 
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( ,.-«rT20N 2: CASS 2N?0RyATI0K' 

C,. R-300 VTRiriCATIOS 



S£K\'IC2S PRWIDSD 

:tKSrJOCr:0NS: ?le&se answer the - 
foilowinc questions (A and B). for 
^»ch cf the ser\*ice categories ' 
(5>-5o) listed at the right " 



SERVICE 



Indicate your findings recarding the 
^ agreement of data found on the R-300i 
or its equivalent, to that found in 
the case record relative to the 
client's receipt or non-receipt of 
the fcliowinor services. ^ Verified 

* Not Verified 

• ~ Case Record Only 

R-300 Only 

N/R 

N/A 



Indicate your findiTics regarding the 
agreement of data found on tne R-300 
or its equivalent, to that found in 
the case record relative to £he cost 
status cf t,he ^liowmc services. 



Verified 
Not Verified 
Case Record Only 
R-300 Only 
^/R 
N/A 
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For auctions 1 throu^ 13, ple*$e inswer with:. 

@ . NOT APPtlCABL? (N/A) 



ELIG. 



IWRP 



SUPP. 
EVAL. 



SECTICN II: 
A: 



• eVAlUATION OF REHAIILITATICH MTEHTIAL 
PR£LIMI/«ARY DIAGNOSTIC STUDY - STATUS 02 



ANSVtR 
YES NO N/A' 



ex) 



do99 zha COM* r^eord » • v 

1. contiin an applicition for VR %tn\Ml 

* 2. document that* the client was ad1vs«d of tht confid«ntia11ty of all * 

Infonatlon pertaining to the client's case? 

* 3. docjment that the client was advised of the need fo^ the client's 

writun consent for the release of information pertaining to the 
case? 

Dcm9 ihs pTtivninay diagnostic Mtudy , . . 



9. 

to. 

11. 
12. 



• »^'ilrsd for confonmnce with Regulations jwd Guidelines. 

# »<ay ae discussed witft .-neillcaT consultant for clarlficatlon/lnterpretitloo* 



0 ® 
O © 

0. © 



0- 



include an appraisal of the current general health status of the 
client? 


0 


©' 




include a psychiatric or psycholofical exanrlnation in -all cases of 
!nenui or enotional disorder? 


© 


© 


© 


include such exairinatlons arid diagnastlc studies as necessary to: 
a. dctenxln* eligibility? 


© 


© • 


© 


b. determine the need for extended evaluation? 


© 


©- 




place orlmary emphasis upon the detenirtnatloo of the client's poten- 
tial hr achieving a. vocational goal? 


© 


© 




sucport the dete mi nation that the client has a nedically recognized 
pnysi^al or nental disability? 


© 


© 




support the determlAatlon that thc'nedically recognized disability 
constitutes a substantial Jiandlcap to eiploymenc for the client? 


© 


© 




support the determination that '/R services my reasonably be expected 
to-senefit the client In Unas of eoployability? 


© 


.© • 


0' 


suoport the detemlnacion that an extended evaluationi Is necessary 
to ^tennine that VR services ntfy ntasonafrly be expected to benefit 
the cljenc in terns of eaployabtli ty? 


© 


-© 


© 


cpnuin itu supporting \dcten14natlon that the client is severely 
aisaoled? 


, © 


© 


10 


document the use of state agency consultation: 
a., mdlcal?^ 


© 


© 


. © 


b. psychiatric? , 


© 


© 


0, 


c. psycfwloglcal? / . - . 


0 


© , 


© 
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SECTICW 11: EVALUATION OF REHABIlltAnON POTENTIAL 

^ A. PRELIMINARY DIAGNOSTIC STUDY - STATUS 02' 



MH: IMPORTAIfr INSTRUCTIONS • ^ . 

-if u$k Is certified e]ig1l?le^for rxtanded evilMtion, go to Section HB, ques- 
tion 14, page 6 to cocplete subsection on Exten<jed Evaluation. 

if case is certified eligible, go to Section III, question U'page 13 to cowplete 
section on Eligibility* " 

If case data does not support the detenni nation thatythii:.^ £lient'^is eligible, 
\ although the client is certified as such, set cas^side forrwi^by full rer 

j view tearn* 

i ' * I 

I ^ J 

If case is closed In Status 08, go to Section page 23, to complete the 
aopropriate subsection on Status 08 closures. 



; 



f 



\ 



ELIG.- 




EVAt. 
























































X 





/for ite«s 14 through *2, pleast Mjw«r with: * ^ 
@ • NO 

(D « NOT APmCASLE (K/A) 

SEckcH in EVAIWTICN OF REHAIILITATIOK WTerTIAL 
8. 'eXTL»<D£0 EVALWHW- STATUS 06 v 



YES «0 K/A 



. Certfflcaticft for Extended Evrlmtion 
Do€M :tm cm4 record . • . 

♦ 14. contain a certification for txtended evaluation to deterstlne rchiblllta- 

tlon pountlal? * ^ ' * , 

Xj%yES* do€t tH$*ctstifieation for Mst^rtdtd waUtatian • . . 
*a. Indtcata the date of certification? 

•U. include the sf?ha;ure of the aporoort'att state agency staff s«3w 
*btr (designated certifyirtg agent)? 

*c* indicate that the client has a«t the PMsU^^ requiremnts? 

d. 'Indicate the' client's naae? 

"shew tnat certification for extended evaluation was based on: 

•a. the presencs of a »d1cally recognized physical or ntnUl disabi- 
lity? 

^. 

•t). the deterainatlon that the fliedlcally r«cogn1ied physical or »enUl 
disability constitutes a subsUntlal handicap to enployuent for 
the client? 

•c. ithe Inability ta lake a deteniinatlon that '/R services «1ght bene* 
fit the client In cams of e/nployabllfty without providing an ex-, 
tended evaluation? , 

^ 16. show that the extended evaluation period began with the date of certifica- 
tion to deteralne renablHution potential? 

• 17. Show that<extended evaluation ms provided to the client for a total period 

'not In excess of 18 months? 

!ndW^duaM;*4 'Writteff^ilehablHUtlori P^qrut (IVRP) 
'or £;ttenge^ £/g/uatt0f! 



® © 

® © 

® © 

® © 

© © 



18. 



!s the^an IWRP for extended evaluation in* 



the case record? 



jWTE: IWItTAMT WSTROCTIOKS 

jlf there Is an IWRF for extended eviluatioii 1ft the case record, ansi^r <<uest1ons 
jlJ wrsugft below. ^ ■ . „ 

If a>er- ts no IVl* for extended evaluation In the cas» record, 90 on to questtoo 
page lO. 



« !iequirta far csnfpnanct Je^Utions iM oitidellnes. 



■ \ 



o 
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(D 



® © 
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0 ® 


'© 


d) © 




0, © 




©,© 


! 


(D © 






ELIG. 


IWRP • 


SUPP,, 
•EVAL. 








X 


X 






X 






X 








(X) ■ 






(X) 






(X)| 






. (X) 
















X 






X 








(X). 




X 






X' 




* 


■ X 





ERIC 



SECTIOail: EVALCATIOH or 'REHABILITATION POTEKTIAL ■ 

8. 'EXTc^iOEO HVAUJAT10N *.STATUS^C6 (continued) 



YES Nd 



1 

J 

i 
i 



Individualized WHttgn ?^h«blHut1on Prqgraa (iVXy) ' 

For ixtence^ evt^uagion ^ ^^^^ ^ ^ > * ^ ^ 

tht iWP /«r MSUndMd mluatich (szat* fcnO . ,^ . . '< ' 

* 19. present thty^entral buls for « deUnrfnAtion that en extended evaluation 

of rehabilitation pountlal Is' necessary to mk% M dctenrfnatlon of ellol* 
billty? • ^ 

* 20. set forth the tcnu and conditions for the provision of service, Including: 

*a. client responsibilities In carrying out the pro^rtn, such as atten* 
dance,, cooperation, etc? 

•b, the extent of client partlclpetloo In the cost of «rv1ces? 

♦c. the extent to vhlch the client Is eligible for sinller benefits 
^unoer other prograns?* 

/ ^ * ' 
^, ^the availability of VX funds? 

> 

e. the availability of openings at facilities and schooU? - ^ 

f. the possibility of* delay In a phase of the progran? 

g. the provision for changes In the progru due to new Infonnatlon 
and changing conditions? 

> 

h. the condition that the continuation of services is dependent upca 
an on-going evaluation ( including the health status of the client} 

fh^mjj^tlihood of >the client's ability to benefit in tents of 
ployability? ^ 

21. doctjnent that th« client t«s Inforaed^of client^ right;! and reoedies, 

including: « ' 6 . 

♦a. the -right to be-Jully consulted regarding ipardMnges or anendnent^ 



in the renaoiHt 

'*\>. the right to adcrin? 
services? 



Tve review in cast 



o# dissaics 



faction 1*1 



•c* the av«i1eoi4i:y of resources ifithin the Client Assisunce Project* 
^ "nhtrt appropriate? 

\ ^ • ' 

♦d^^tfte rigftt to participate in the annual r«viewof*th« prograie? 

•e. t^e right ^to' participate In* tht annuaT review of tfw ineligibility 

V .decision? , ' * 

* 22. reflect that the VM^P fot exUnded evafuatlon'Ms arintiined.as e, separate 
part of the case record? \ 

* fteqmred for confonvnce with Reflations inct Galdtlines. ^ ' * ^' 
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EVALC 
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IWRP 


SUPP. 
EVAL . 
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X 
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Y 












I 

V 
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1 




Y 






Y 
A 




i — :^ 








X 




X 


X 


(X) 




X 






X 






X 






X 






X 






X 






SECTICH III 
8. 



tYAlUATXCH 0Fil£KA3ILITATI0N POTLSTIAL 
EX7E<K0 iVflimiQH • STATUS 06 (contlnyed) 



T£S 



.HQ / 



DC9S : 

• 23. 

• 24. 

• 25. 

• 26. 

• 27. 

• 23. 

• 29. 

• . 

• 3T. 

• 32. 

• 33. 

• 3S, 

• 36, 
» 37. 

• 38. 

- 39. 



■ ' 
s.*iOM t?iit tA« clfertf^rfcxlvtd « ccpy of t.n« 2VRP sudsuntiil laerrd* 
ants? ^ 

show t'lAC tht progru InftUted <ft«r tht ^rttflcAtlon for extended 
cviluitlcn? . * 

shoM Lilt tht progru w«s continuously drvtloptd <fttr the ctrtlficitlon 
for txt«nded tviluitlon? 

Indicate th«t the progrsM iras developed and a£ended with the client's 
pirtlcipatioo, or as appropriate, with the parent, gujrdiin, or pther re- 
presentative? 

show thkt consideration wis given to the client's need for assistive devices 
or supportive serviced In the developrant of the progrsn? 



state the Interaedlite rehibllltatlcn objectives? ' ^ ^ 

state the VR services to be provided whlcii art necessary for the detennlna- 
ticn of renaollltatlon potential? 

contain the projected date for the Initiation of eich service? 

contain the <nt1c1pate(l duration for etch service planned? 

provlce tne projecud t1» within which rehabilitation objectives aay be 
iciTeved? 

Sftcw nat a thorough asseisrtfnt of ttre client*! progress was mde at least 
once in every SC-cay period^durfng the provision of servcces under the 
exteficed evaluation? ' . 

state the objective criteria upon which in evaluation of the client's pro- 
grtss is sased? 

suu t.ne procedure by wnicii the client is evilutt^d? ^ 
contain a sc^dule for the periodic reviews end progress evaluations? 

ccntain a record of the results of scfteduled reviews ind progress evalua- 

ticns? 

sncw tnat a fomal , annual review has oeen conducted if th€"lWRP fias 
acniaved at least first anniversary status? 

dccvnent the client's views, or, as' appropriate, the views of the gartnt* 
guarsian, or ot.*ier reoresentaeive concerning the objectives and 'i% ser- 
vices deing pr^svlded? 



mi\ lMPCR7A/ff I.HSraUCTICNS 

If t.*:e cMiint las seen detenrined eligible as a result of extended evaluation, 
answer qijestlens 40 and 41. / 

t.ie z\*t^t nas deen 2eter»ined ineligible as a. result of extended eval ua?^ 
t:cn. 'answer question 42. / 



/ 



* ^ouirea 'or conformnce with (^gulations and duidelioes,. 
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IWRP 


SUPP. 
EVAL. 
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* 

X 






X ! 




X^ 




X 


X 





SECTIO; 



r/ALUAHON OF R£W8ILITATICH WOTIAL 
iXlViOEO r/ALUATtOH « STATUS Oi (continued) 



, AKSVW 



I 



.Tom zhM x»i rtcord ... 

• 40, conuin 4 c<rt1ffC4Cion of eligibility for tht continuinct of VX scrvitts7 

conuin in tsKR^mnt to Xht progria «ntn the exttncitd eyaluation progru 
is urairvteo otCAuse the client is oetenrintd eligible ? 

^ *Y YES, coji record . . . 

*4. shOM that the wended IVRf set forth all new decisions and facts 
not already covered in the progrue? 

•tf. show that the a«nded IW set forth planned services not already 
covered in the prograo? 

* iZ. contain a closure sxatetnent as an anndaent t,o the prpgran for the jclient 

tfetirained fneligibU because of a lack of renabilitatipn potentiaJ? 

•/ TES, ifftf* ihs csjt r^csrd . . . 

*a. show that the decision to teminate. services iras sude in full ceo* 
sultat.tfln with the client,, or as appropriate, with the parent, 
guardian, or other npres/ntative? 

*b. sho« that the raticnale fdr the decision to teminau services 
was reprcco as a'certf ffeVaaen^tnent to the If^ for extended 
evaluation? 

*c. Show that a certification of ineligibility was then executed? 

*d. snow that the provision was ^de for t periodic review* at least 
annually, of the ineUgibility decision? 

* Squired for conforaance witn Regulations and Sufdelines. 
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S£CTIC?< lU fVALWTICN OF SfflASIirTAnCH ?OTU(TIAL 
S. EXTiNOEO EVALUA7ICH - STATUS 06 



Jell very of Servlcw 

INSTajaiOnS; ?H?ast »ns> 
follcwiR5 questions (A-S)N 
of tht strvi^ cate^ortts 
Usttd at the rt^ht. If the 4n- 
utr space is dlanld, please ^ on 
to the next question for that 
service. 




SERVICE 



*A. Does the cise record Indicate tiat 
the serrlce was nectssiry to «ter- 
sine the client's rthi^flltatlon 
pountlal? 

•3. toes the case record docuvent that 
tne service was planned /or the 
client? 



•C. Does tne case ricord decent tJiat 
ti'.e service was given to the client? 



If til , 

•0. Was the service provided consistent 
wl!n tte Indlvlduallzfd written 
renaollltaticn prcgraio for extended 
evaluation? 

♦E. '<^a$ full ccflslderatton given to any 
slailar seneflts avallaol* to :.ie ' 
client to :5e«t. In -nole or In part, 
the cost of the service? 



ifas the service given only »fter the\ 
cHent'f fiflancial sutus was evalu- 
ated. If the state rsgulatlons re- 
quire a 'neeos test" prior to fur- 
nishing t*!e service? 



•tj. was th« quality of case wagewnt 
In the delivery df the service ad»- . 
quau? 



^ fiwutr^d for confomince wiGr Regulations and Suidelift«. 
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Ted 
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NO 

(S) 

N/A ^ 
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SECTIM U: r/AlUATIOH 
. 8, SXTLH0E9 £V 



r/AlUAfrOH Of ROUSlLnATIOH WTEiaiAL 
SXTLH0E9 CVAIWTIOH - STATUS 0« 



I 
I 
I 

J 



Oell /try-Qf Services 

lHSTXuaiO^'?lea$« answer th» 
foHcwfng quesclOM (A«C) for escft 
«f cnt service caujorles (45-54) 
llsUdat the rtsnt. If the an- 
swer space Is bUnJe, please 90 on 
to the next question for tiat 
serylce. 



SERVICE 













1 

(X) 















•A. Does y)e cise record Indlciu that 
the service was necessary to deter* 
■Ine the client's refiabl11t4t1on 
pot«nt1aI? 

*8. Odes the case rtcord docunent that 
y,e service was planned for the 
client? 



•C. Cees the case record docujaent that 
the service was given to vie client? 

\ 

If YgS . 

•D. Was the service provided consistent 
with the Indl /IdualUed written 
rehaollltatlon prograa for extended 
evaluation? 



rfas .full consideration given to any 
slolUr oeneflts ava11at>Ie to the 
cli-^t to ;<»«t. In wnole or In part, 
the cost of the servica? • 



•F. Sias the service given only after the 
client's financial status was evtlu- 
aUd, If the s^au regulations- re- 
quire a "fleeds test* prior to fur^ 
nishtng the service? 



S. Was the Qu«11t/ of cas« aanagefient 
,1n the delivery of th» servlct edt- 
quau? 



' Quired fdr conforranc* witft aeguUtibos and Suldellnej 
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SECnON II: EVALUATION OF «HA8ILITATI0H POTENtlAL 
EXTEHDEO EVALUATION • STATUS 06 



NOTE: IMPORTANT INSTRtlCnONS \ 

If case is certified eligible, 90 to Section III; question 1» p«g€ 13, to com- 
plete section on Eligibility. 

If ci'se .data do,es not support the determination that the client 1$ eligible 
although the client fs certified as such, .set case aside for review by full 
review team, ^..--^ 

If case is closed jn Status 08, 90 to Section VII, page 23, to cowplete the 
appropriate subsection on Status 08 closures. 
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for qutstlons 1 and 2, ptus* uantr withi 

CD • NO ^ 

@ • NOT A7PI.ICABU (VA) 



'. ) 
ELIG. 


IWRP 


SUPP. 
EVAL. 


X 
















' — • 1 


X < 








1 . 1 






(X) 1 



SECTIW III; aieiSILITY • STATUS 10 




YES XO K/A 



ERIC 



:>9«« ^ edt4 r€OOTd . . . 
♦ 1. conuin a ctrtlflatlon of eligibility? 
Zf YSS, ^ certification of •li^ibxliTa • . . 

*)|. Indlcatt tilt dau of ctrtlflcatlon? 

• b. irtclude the signature of the appropriate state ageniy stiff 
mcer (oesisnaud Ctrt1fy1n9 a9ent)? 

i • c. Indicate that the cl^c has iset the iasic eligibility re* 

quirefflents? 

d. indleau the cllent^s nasK? 

* Z, contain data supporting^ determination that the client Is severely 
disabled? «• 

* ^^ulred'for confoHnance with ftegulatlons an^^ldtl^nes. 



U2 



■ N 



-13- 



CD ® 

® © 

0 ® 

CD © 

CD 0 ® 



Forqutstlons 3 through d> pltist iftswtrwith:' 

0 • L£SS THAN AKQUATH 



ELIG,. 



- IWRP 



SUPP. 

EVAL.. 



S2CnO« HI: EUCIIIUTT - STATUS 10 (cofttlnut4) 



USS AOq. H/A 



Phytiqy or Wenul DisabHlty 

50U ucl^ <£0«« tKg ocwlcf dcc\mmtaz{cn vt^ ths cats record . . . 

• 3. tsublUh the present of a physical or nenUI .disability with 

ntttssaiy ntdical, psychiatric* psychological, and other infor* 
Mtioii?> 

^» show that tht physical or wnul disability Aurlaljy ti»its, 
coQtrlbutt^ to Halting, or if not correct^, would probably 
result in Halting Ir client's activities or functioning? 

SufasMntlal Handicap to Egployatnt 

Scu iMll do4m tht ccimMttoT dpcmiti itazi cn in thm cats rtcord . • • 

* 5. analyze the specific ways in which the following factors, as 

appropriate to the client, iipedc the client's occupational 
perforaance by prcvcfiting the client fro* obUining, retaining, 
or'preparlng for eraloyioent consistent with the client's capa- 
cities and abilities: 

♦a/ Mical factors? 

^ 'b. psychological factors? - -^^^ . 

*c. vocational lectors? ' 

*d. educational factors? 



show that the related factors which bca 
vocational partidpatioo w#re considered? 



J- 

r upop^succ 
d? f 



successful 



* 7. 



show that the substantial handicap to eiployment exists, even* 
though the client fs efl|}loyed, because the client is unable 
to cbuin a gainful occupation cdnsistilent with the client's 
capaci ties' and abl 1 i ties? 

Seasonable Exsectatlon ' . ' • 

3ov tjtlZ dots tht ecmtsiar dacimsKtation i/t ths cats Tttford • « • 

3. shM the likelihood of VX services enabling the client to 
achieve vocatlortal goals consistent with the client's cap»» 
cities and ''abilities? 



A' 



.Quired for confonancs wittr Regulat1of?i and Guidelines. 
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for QMsefons ^ througn 7, pleise insw«r with: 
© • YES 
0 » NO 
' 0 • WT APPLICABLE (N/A) 
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SECnCN IV: EYALUATIOfi OF REHASILITATION POTEJCIAL 

THCflCUGH DIAGNOSTIC STUDY - STATUSES 02 ANO 10 



YES .VO ♦ • K/A« 



♦ 1. dtUmiiw the niture of VX strvlces netded by tht client? 

nydetf to attain the 



2. detenrine Che scope of VR services which are 
vocational goals of the'cUent? « 



*l 3. contain the necessary specialty exaQln<t1oh(s):| 



Zf YcS, iots z'n* 'Tharough diAqne$tic jrudy dacitimnt ^'"^ I'MLJf — '~^r 

«f a. a visual evaluation In all ci5es of visual liroalment? 

b. a scrsening for hearing loss In all cases of blindness? 

*# c. an evaluation of the auditory systeo In all cases of 
/ hearing lirpainnent? 

*# d. a psychological evaluation In ill cases of cental reurda- 
elon (Inducting a valid test of Intelligence and an assess- 
nnt of social functioning, educational progress, and 
achleveoent)? 

e. other?^ Please specify: . 



*# X, docuraent that when planning for physical and mntal restoration ser- 
vices, consideration Is given to the following criteria for the pro- 
vision of physical and nental restoration services: ^ 

•# a. t*iat the clinical sutus of the physical or Mntal condi- 
tion is stable or slowly progr^lve? 

V b. that physical or aenul restoration services are expected 
to elimlnace, substantially reduce or contain the lopact 
of the disabling condition within a ^reasonabU period of 
tine-? ■ 

•f^. cooslst df a cccorehenslve evaluation, to the degree nnded, of per- 
tinent aedfcal, psychological, vocational » educational, and other 
related factors which bear dn the client's handicap to cf^loyMnt? 

• 6. conuin dau supporting a oettmlnatl on that the client is severely 
disabled?! - ^ ^ 

7. docunent the use of sute agency consultation: 

' a. wdlcal? , 

b. psychiatric? 

psychological? 

* ^qutTYd for confomance with- ^utaclons and Guidelines. 

# Hay 3e discussed <«1tn »d1cal consultant for clarlficationr/lnterpretadon. 
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for questlcni fif throu* 



' ADEQUATE " 

Qj: M im APPuiCAau (h/a) 



r 


(wRP ; 


fiVAL. 


• 




fl 

< 
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ScCTICN IV: EVALUATION OF «£HA8IUTATICH POTEJCnAL ; / ^ ^ 
, TWROOSH OIAOfOSTk STUDY - STATUSES 02 X» 10 (continued) 



A. To thM*dtqTt9 n^t—^j, hoo iMll do*M ths daxa 'in thm eas* rtcerd 
dM9dTib€ . • . ' ^ 

in* tAM cos* r^icrd ojui ^ ^C sm . . . , 
I 

• 8. perscnal adUusfbent?' 

• 4, vocitlonal adjustjnent? 



« 10. social adjustsent? 



• 11. Intel lig«nce level? 



* 12. prior educatlorul actiievetnents? 



• 13 work experleqc^ 

* 14 * ibiiizy to acquini ^ccapatfonal sicill? 



15. capacity for $ub«Msful joi perfomanca? 



« 16*. client's esoloyiM/opportui^ttftS? 

i 

• ae<ju»rea for confbraance with fle^uUtlofa and Guidelines. 
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445. 



ANSWER 
LESS \ADEQ. M/A 



A. (describe) 


0 




0- 


8l (appraise) 


®^ 






A,%Mcribe) 


0 • 




© . 


B. (appraise) 


0 i 




© , 


A. (describe) 


0" 


© 


■ © 


,-^..5. (appraise) 


0 


© 


© 1 


(descHbe) 


0 


© 


© 


6. (appraise) 


0 ' 


© 


© 


' A. (descHbe) 


0 


© • 


© . 


3. (appraise) 




© 


.© 


/. ^(dtscHfce) 


0 • 


© 


. 0 


"^'B, (appraise) 


0 


© 


0 ; 


A. (describe) 


0 


© 


© 


^B. (appraise) 


0 ■ 


© 


© 


A. (descHbe) 


0 


© • 


© 


B. (appraise) 


0 


© 


© 


A. (^cHbe) 


0 


•© 


8. (appraise) 


0 


© 


© 



] 



for queitlom 1 Uirough 2S, ptcut inswcririUi:. 

® • Its 

© » HO 

® • KOT APaiCASU (K/A) 



ELIG. 



IWRP 



SUPP. 
EVAL. 



ScaiON V: INDIVIDUALIZED mVlVi RSWSILnATJOH PROGWM 
(IWRP) • STATUS 12 AND ABOVE 



YES NO H/A 



• 1. 1$ there 4n IWRP In th« case record? 



NOTE: IMPORTA^^T INSTRUaiCKS 

If Were U an IMR? in Wie ci$e fecord, ins^er questions 2 through 26 belo*. 

If there 1$ no I^RP In the c*$e record ind services »r? being prov1<Jed, go 
to Section VI, page 21. 

If there Is no IWPJ* In the case record and no services are being prtjvtded, 
;o to Section VII, page 23, 



CX) 



• 2, present the ^neral basis foV a deternri nation of eligibility? 

• 3. set forth the tenrs and conditions for the provision of serv1cas» 

including: 

* a. client responsibilities in carrying out the prograM, Such as 

cooperation, attendance, eCc? 

♦ b. the extent, of client participation in the cost of services? 

• c. the extent to *ft1ch the client is eligible for similar 

benefits unoer otner program? 

d. the ayanabilit/ bf VR funds?. 

e. the availaoility pf openings at facilities and schools? 

f. the possibility, dV delay inja phase of the prograai? 

g. We provision firr changes fn the ^progrijp^due to new infor- 
mtion and changing cbnditions? [^-^^ 

h. the carfdition that the r^ontinuatiort of V*rvice$ \% deoen- 
.'dent upon on-goirtg evaluation (including the.*healtn sutus 

of the client) and tne likelihood of th< cliene's ability 
to benefit in tens of e«|}Ioyabf ii.ty?, 

• 4. docunent that the cjlent MS.Infomed.'of cli^nt^gh^ and re«dies 

. including: ' \ ^* 

> ♦ '^ y > \ 'i 

* a. the right p be fully consulted regarding 'tiff cianges or 

, ^^pendnen'ts in Vie rtnatfilitatlon'progna? f". 

b. the rtgnc to Adanmstratf ve rrirt^i .Iff case>of di.ssatisfic* 
tlon «#itn services? *.* * 

• c. the avai'aotltty of resources wftiiin t}»e/Cl-ifWt AssiJ^^nce 
^ giroject. -nere ipprcpriite^? -* > ' *' ; ■ 

* i^ut red for conforinance ^liffl^gulatfon^.snd ^i^iin^v . •> , 
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BUG. 


IW^RP 


SUPP; 

EVAL. 




. x' 






Y 
A 






X 




• 


X 


















Y 
A 












' X 






X ' 






X 






X 






X 






■ X 






X 






x\ 


\ 




X . 






X 






X 






X ' 




< 


X 





SEaiCH I?ftlVIDUALl2E0 WRITTEN REHABlLtTATlW 

(WRP)'- STATUS 12 ANO ABOVE (continued) 



YES 



AHSKER 
NO N/A 



Am« tht mP (9tav* form) , . . ' 

• 4, ' docunwt tftat thf cMftnt m4$ Infonred of client rights ind-rtwedits 

. incVoding: ' 

♦ d, tht, right to p«rt1cip*U in tht wnuil r«viw of th« progr*«? 

♦ t, the right to parti cip*tt in^he annwl review of the ineligi* 

blllty decision? 

5., reflect xjtin the IWRF was utlntalned as a separate part of the case 
record? 

D0€» th$ ZVFP ... 

6. show that the client received a copy of the IWRP tnd substantial a«nd- 
oents? { 

7. show that the progran w« inituud-after the execution of the certlfi- 
citlon for el1g1btl1t»f 

8. show that the progn«\was continuously developed after the execution 
of the certlflcatlort fO<;el1gibf llty? 

9. indicate that the Jrognn was developed and awnded with the client's 
participation or, as appropriate with the parent, guardian. Or other 
representative? 

10. show that consideration was given to the client's need for supportive^ 
services or assistive devices In the dewelopjaent of the prograu? 

11. place primary eaehasis on the ^determination and achleve-oent of a 
vocational goal? / 

12. state the Ipng-range employment goal? 

13. state the Intemedlate rehabilitation objectives? 

14. state the specifU VR services* to be provided to achieve the Inter- 
.nediate objectives and the eiroloynenc gdal? 

15. conuin tiie projected date for the Initiation of each service? 

16. conuin the anticipated duration for each service planned? 



17. 

18. 

19. 
20. 
21. 
22. 



provide the projected tine within idtich rehabt.^1 ution Objeotlves 
and goals may be adtleved? 

stau the objective criteria upon »<h1ch an evaluation of the cll^ 
ent's progress toward an 'e«ployao111ty goal Is based? 

state t*)e procedure by Mhich the client Is evalu«Ud? 

conuin a schedMle fOr the periodic reviews and progress evaluations? 

conuin a record of the results of the scfteOuled reviews and evaluations? 

sn^ that a fomalVsannual review has been conducted. If the IWRP nas 
-icvievea at-le4"st-fi^Tt-annlver$ary stat^asj _ _ _ 



23. docunent the 'clfent's views, or as aporooriate, the views of the oarent, 
guarxUn, or other reortsenucive concerning the goals, objectives, and 
'/« services Deing provided? • ^ 

• ^equWo for confomincs i^itn aegulatioos and Suldelioes. 
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^^^^^ 



CI Tj^ 

fctlO. 


IWKr 


SUPP. 
EVAL. 




• 

V 

A 


• 










X 




'* — — 


X , 


: 




X 






X 






X 





























_S£CTICIW V: irarVIOUAUIZHO WHITTEH RaABItHATlOH PROGRAM 
(iWRf) * STATUS 12 A80VE (cdnClnucd); 



HOTE: IWRTAM" IftSTRUaiOHS ' 

If cut Is c1ost4 non*rthabniut«d, ansi«tr question 24. 

If cue Is closed rehAbllluted, 4ASMr question 2S. 

If CISC Is closed ret^ebll luted en^post-enploynent services have been 
planned* answer question 26. 



Do€» thM D/SP . . . 

* 24. contain a closure statement as an anendnent to the program when the case 
Is closed non-renai>111tate4 because the client Is not Capable of achiev- 
ing a vocatlcnal goal? 

If YES* do*M ths aa3€ rtccrd ... 

* a. show that the decision to terminate services was nde In full 

consultation with the client or as appropriate, with the parent, 
guardian, or other representative? ^ 

* b; shot that the rationale for the decision to temlnate services 

was recorded as. a certified aoenonvnt to the IWRP? 

* c. show that a certification of Ineligibility was then executed? 

* d. show that the provlslcn was nade for a periodic review, at 
^ least annually, of the Inellglbftlty decision? 



J 



Do4* ch* rJItP ..." 

* 25. contain a closure statement as an aMndoent to the program for a case 
closed nnabll luted? 

If YES, dagM zhs stazsimnt inaluds: 

* a. a description of the basis upon which the client was detenrlned 

to be rehabinuud? 

* b. the type of occupation? • . 

* c. the length of tliai the client has been engated In the occ^a* 

tl6n? • ' . 

* d. a description of vhyi* It Is detentrined that the client sade 

satisfactory vocatlonil adjustment? 

* e. any plans for the provision of post*e5|}1oyA«nt servlcts 

a suitable objective has been achieved? 



* Quired for confomtnce witn lleguUtlons antf Guidelines. 
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YES- 



AMSUER 
NO • f|/A 
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I 
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ELIG. 


IWRP 


SUPP. 
EVAL. 




X 








(X) 






(X) 






(X>* 





















SECTION V: INOIVIOUALIZED WRITTEN REHABILITATION PROGRAM- 
(IWRP) - STATUS 12 AND ABOVE (continued) 



Se^^ices? 



' * 25. I» there »n umniti IWH? for j^ost-Employnent Se\vices? 

Jf YES. do€s tht amdd mP . . . 

provide the wis on trfilch pUns for 'post-«Bp1oynent sen^lces 
h«ve been developed? 

describe the outcoae to be achieved or the outccne of post- 
eaiplo>!nent services? 

^ *c. describe the type tnd extent of post*«:p1oyinent services to be 

or being provided? 

*d. Include the progress tssessmnts? 

*e. dximent the reason for the tenolnetlon of post-esoloyment ser* 
vices? - 

*f. document thet the decision to tera1n«te post-efrployment services 
»MS made In consultation with the client? ^ . 

* Required for conforaance with Regulations and Guidelines. 
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YET NO H/A 
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ELIG. 


IWRP* 


SUPP. 
EVAL. . 
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i 




■X i 




1 . ! ■ ■ 

! X i .(X) 
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(xr. 






(X) 
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SECTION' YI: OaiYEilT OF SKYICES^ STATUS 14, 16, 18, 20, 22, and 32 



1 


2 


3 t 4 


5 


•5 


7 


Evaluation and 

Olaynostic 

Services 


e 
m 

VI «« 

IS 


1 

u u. 
o 

VI <-» 

la 


e 
o 

u 
« o 
** 

e VI 


e 

Si 

3 U 
« V 

u ^ 
o •» 

> o 


« 

u 

m ' 

C 

»* 

i 


S 

« 

o 
& 
v» 
e 


1 


SB 


TCS 

f ■ 




® ' 




yS 
® 






IS 

YES 

t 








© 1 
Y5 


® 




f 

® 


ss 
f 




SB 
® 

in 


s 1 

m 




T^ 


ySs 


t 


i. 


i 


Q 

Ys> 
® 






Si 


(T 

YE 

i 


s 

) 


/TV 

f 


. yIs 

f 


yes ' 

f . 






1 


(i 

yI 


f 


Y§ . 


ss 
f 


i 


® 






yD 

® 
li/A 


1 sg 


yI^ 

®- 




® 


(3) 


i 

NO 






f 


i ■ 








yB 




® 

HO 


i, 1 



I 



IHSTRuaiC'C; Pleiit aAS»«r tfte 
, foMcwing questions (A-l) for each 
of the service categories (Iteias 
1-7) lUted at tfte rignt.^ If an 
answer space Is OlanH; pleaje go 
on to the next question for that 
service. 



HEM 



SERVICE 



♦A. Does the case record indicate that 
the service was necessary for the 
- client? • 



•5. Coes the cue record docyoent that 
tie service was planned for the 
client? 



•C. Occs the case record docunent tnat 
the service was given to we client? 



If YES. 

^/ 

•0* Was the ser/lce provided consistent 
with- the diagnosis and evaluation of 
renaoilitation potential? ' 



Was the service provided consistent 
with the Individualized i<ritten reha- 
"SiUtaticn progrta? 



Was tlte service provided consistent 
W1UI e«^)loyinent at closure? • 



Was full consideration given to any 
sioilar benefits aval table .to the 
client to -neet, in whole or in part, 
th« cost of the service? 



*M. Was the servict given Only after the 
cHenCs financial statuj was evtlu- 
ated.' if tne.state rsgulaticns rt- 
qutre a 'neet^ ttst" prior to furnish- 
ing are service? 



t. *.;as the quality of cis* wnage-wnc i!» 
the ceijveiv of the servtce.^icequatt? 



/ ^equirid ^or conformance '•ith ^gu Unions and 



Suioelines. 
-2U 
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ELIG. ■ 


IWRP 


SUPP. 
EVAL. 




• 


i 




X 


m 




X 


(X) 














• 




fX) ' 






(X) 















SECnON VI: OaiVERT OF SEJWICES - STATj^S- U, 16, 18, 20, 22, *nd 32 





ITEM 


8 


9 1 


JO 


It ] 


12/ 


13 




lH5T!nx:ilO.HS: ^le«t myntr the 
following questions (A-I) for e*ch 
of tii9 service esU^oHes {Itt^ 
^U) listed dt tte Hgnt* If in 
instter space Is blink, please 90 
on to tAe next question for that 
senricer 


raVlCE 

•\ 


Services to the 
Family 


kl ^ 

*<» 

V U VI 


S 
« 


^ 0 

U U 3 


^ « 

0 

at 

m 

0 m 


*» 
c 

{( 
«i 
u 
n 


e 
0 

.'I 


•A, Does the cise record 1n<jicit« thit ^ 
the service necessary for the 
client? 


i 


yIs 
no 


i 

•HQ 


^- 

NO 


SB 
.© 

NO 






*8. Ooes the case record docuisent that 
the senrice was planned for the 
client? 


1 


i 

HO 


® . 

NO 


f 

' NO 


1 

NO 






♦C. Ooei the case record dccuineni that 
the service 'was given to we client? 

If YES, ' 


1 

®. ■ 


1 , 


© 

YES 

© 

NO 

® 

N/A 


f . 

N/A 


,<g 
® 

N/A 


N/A 


® 

•VA 


•0. Was the service provided consistent 
witn the diagnosis and evaluation of- 
rehaolllutlcn potential? 


Q 




© 

NO • 




f 


y3 
®. 

M 


0) 

YES 

© 
• SO 


*t. Was the service proy1<Jed consistent 
with the Individualized written reha- 
blliutlon prograa? 


® 

YES 


® 

YES 


© 

its 

NO 


© 


© 

YES 


Yw 


© 

YES 

© 
HO 


•F. Was ^he ser/lce provided consistent 
ititi\ esoloynent at closure? 


® 

NO 


1 


© 

NO 


© 
yS 


yS 


® 

Yd 


id 


Was full consideration given to any 
similar benefits avallao«e to tne 
client to «et. In ^nole or In part, 
the cost of the service? 


yD 

'? 




© 

YES 

® 

N/A 


1 

© 

N/A , 




© 


© 

H/A 


•H. Was ,t»ne service given only after tne 
client's financial status was evelo- 
ated. If W* stAte regulations re- 
quire « 'nee^a test* prior tS furnish* 
fng ser/lce? ^ 


® 

NO 

®. 

H/A • 


? 


s 

f 

■® 

N/A 


i 

HO 


© - 


© 


i 


* I/Was the quail r/ of case aanagejaent in 
the 4*1 i very of tae ser/ioe acequau? 


(i) 

® 

to 


i 


f 


sg 
f 






•i i 
ri 



• Required for conforwrce with .Regulations iiid SuioeHoes. 
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SECTIOHVII: TSRMIJiATlON 0? cases 



NOTE: IMPORTANT IHSTRUCTIONS ' ' * ' 

Pltase prodede to' the appropriate subsection according to the case status at 
closure as indicated belov* .If the case Is in an open status, 90 on to Section 
VIIJ, page 33. . . 

If the case Is closed in Status 08 from Status 00/02 for intervening reasons, ^ 
answer questions 1 through S, page 24. * 

If the case is closed In Status 08 fnxn Status 00/02 due to ineligibnity, 
answer questions 6 through 16, pages' 25-26. 

I If the case is closed in itatus 08 from Status 06, answer questions 17 through 

Zrroiqe 27. . ^ ^ < ^ 

If the case is closed in Status 30 or Status 28, answer questions 28 through 
41, pages 28-29. 

If the case is closed in Status 26, answer questions 42 through 58, pages 30-32. 



238 



For questions 1 throi^ 5, pitise inswr with: >^ 
0 • YES 
© • « 

(3) « HOT APPLICASU (ft/A) * 



ELIG. 


IWRP 


SUPP-. 
EVAL. 


ScCTIOM VII; TEiy^IRATION OF CAScS - STATUS 08 


YES 


ANSVER 

flu 


If/A 








Case Closed Status 08 Frea 00/02 • Inurvenina Reasons 














Vo49 ths coMW record . . • " 

• 1. contain a copy of t^e Case Service Report piioo or the €<;u1valent state 
statistical reporting fora? 














® 












I J Tc5, 


® 


© 










♦a. Is the C4se Service Retwrt R-300 properly comoleted? 






1 


•5. Is the Case Service Report R-300 dated? 


® 


© 










♦d. Is the Case Serrfce Report R«3C0 signed* - - 


® 


© ■ 










♦ 2. dociFient ipeciXtc reasons for the closure action? 




© 




X 






* I'^ y£S, iic€s zh€ coMt rtctKri iicctentrtt that thM alimt xmamilabU 
for inzmpw, rjaluazUn,'^cnd du^wtia afjcUw, or athtr 9€mc€9 M^n* . 

•a. died? . • - - . ' ' •; 












(xr 


® 


© 


© 






(X) 


•b. coved out of suu or was Impossible to contact? 


® 


© 


© 






(X) 


•c. was Institutionalized under circuatances ^ich rendered tse^ctleftt 
univallabUL - . - — . . 






© 






(X) 


' • '3. show that the client was re/erred to other agencies and facilities, as 
appropriate? 

• 4, show that the action taken and outcone were repordl^^to other agencies, as 

ajrproprlate? 


® 


© 


© 






(X) 




© 










X 






5. show ttat tte client, or as appropriate, the parent, guardian, or other 
reoresentatlve, was advised of the reasons for closure and the closure 


® 


© ' *■ 


•©• 






action taJtenr ^ 



^(^..CW TO S£en«-VIH,:-PAfi£"33/ 



*-Re<)u1red for confomance with Regulations and Ciuldtllnts. 
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ERIC 



ELIG". 



-X-- 



X- 



Tor ou«5t1«B» S throu9(i'J6. p1e*se tmvtr with: j 
0 .. IfES ; 

0 » NOT APWCASU (H/A) 



IWRP 



SUPP. 

EVAL. 



SECnON Vllf TESMlHAnON OF CASS - STATUS O* 




(X) 



(Use Closed SUtus 08 Frow 00/02 - I fleliolbnity 
Vc€S ths oa$€ r eco r d . . . 

• ccfitiln « certification of Ineligibility? 

itity 
Tflcatloo? 

•b. inclu<3«''« signature of the appropriate juu agency staff sieaber 
(<iesl5nate<l certifying agent)? 

.^♦c. Include the reasons for the detenrinatlon of Ineligibility? 
d. IndlcaU the client's nam? 

• 7. show that the client does not have a radically reco^ilzed jjhyslcal or nental 

disability? 

• 8: shew that the- client does not have a subsuntlal handicap to eaplcyaent? 

• 9 show that beyond any reasonable doubt the client Is not expected to benefit 

1o terns of e;;ploy ability fn» VR 5erv1ces? 

• 10. csnuin dau supporting the Ineligibility detenainatloo. Including: , 

n. 'a suxary of inedtcal and other case daU obttlned during the pre- 
llsflnary dlagr^tlc study? 

•b. an analysis specl^lng the reasons for the Ineligibility detemlna- 
tlon? 

' ♦c. "a sujwary of ^e counseling activities? * . ^ * 

•dw ddcurnenutlcn'of a review of the Ineligibility deuralnatlon not 
1 Uter than 12 wnths following such oetenrination? 

• Ih show that the IneflglbHity deurarinadon was wdt'only after fuU consulta- 

tion wfth the client^ or ts appropnate, with the parent, guardian, or other 
representative?. ^ > 

• 12. £>cu««ht that the client was notified in writing of the closure action talcen? 

• 13/ docjnent that the client j«s Inforritd In mrlting of client Hghts and reaie- 

dies, including: • > . ' 

th« rignc tt> adtei/tls.trajlve review and fair hearing? 

. *b. the availaaiTlty of resources within tft* Cfent Assistance Pro- 
ject, wnere 'appropriate? 

•t. the HgttC to ^rcicipat« in tne annual review of the fnellgibillty 
detemtnatloh? 



?eou1ni<3 for confbmance wit?T ^guUtlons and Guidelines. 
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YK HO N/A 
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■© © 
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-X 

ELIG. 


IWRP 


SUPP. 
EVAL. 


f 










(X) 






(X) 



SECTION nu THSMIWTICH OF CASES • STATUS 08. (continued) 



TES 



Sc*9 zhg COM* record . 



)4. docmcnc <ny action ind dtcislen involving thtxlitnt*s request' for m 
^ «dsrtn1$trjt1vt review of igency action or f«jr hearing? v 

show thtc ttje client MS ceferreo to other <9encie$ and facilities*, ts 
appropriate? , - / 

16. show that the action taken and outcoae'were reported to, other agencies, 
as appropriate? • " . • 



SO OR TO SECTION VIII, PASE 33. 
i^outnd for confoftiance with ^guUtlons and Ouidetines. 
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'4. 



© ©" 
© © 



© 
© 
© 



''A 



1 



♦ .a 



MOT Am^ICABl? (N/A) 



^elig!. 


IWRP 


SUPP. 
EVAL. 


X 






X 










J 


X 






















m 


X 


X 




X 








X 




X - 






X 


--4^ 


— ' i 

J 


- X 




* 


i 




(X) 






(X) 



SECTias .Yin TERWflATiaS - STATlg~0« 



■1 



YE5 NO . ,.H/A 



\ 



r 



C«e Closed Status 08 frcw 06 - rneliglblllty 
ZAM QaUit Ttcord ... 

• 17. contJin i certification of ineligibility? 

X/YES, <i9c« s^^rSt/Ccacicn 0/ ... 
•t. indicate the dat« of certification? 

•b. include t-he signature of the appropriate $tat« ijency JUff ueaber 
(designawd certifying agent)? ^ 

•c. include the reason^ for the detertrfnaticn of ineligibility? 

d/ indicau the client's naiie?, 

• 18. show that beyond any rtasAiable cJocbf th^ client cannot be expected to 

benefit in tersrf of e^irt^yabili^y froJn services? 

• 19. show that the interrjption of services^'rireicluded ^e continuation of the 

IViRP for extenoed evaluation? • 

• 20. contain the rationale for the ineligibiiify.deteniinati'off as an aand«nt 

to the IWRP? ' • ^ \ ^ ^ 

•21.^ jhow.that the inelj^bflity d«tenination*Vw ««d« only klUr full consul- ^ 
tattcn with the client, or as appropriate *1.th the ^rtnX.. guanjtan, or 
other representative? * > 

• 22. snow that yie IWRP contains the views of the client, or as aoprooriate,^ 

the papent, ^uardUn, or other representa^v* concerning the ineligibility 
decJTOnation? ' . ' 

• 22. doctiaent that the clieflt was infomed fn wT;iting of the closure action taicefi? 

• 2*. docaneflt any action and itfcision involving the client's request for an 

adsilnistrative review of agency action or fiir hearing? 

• 25, * 

• 26. 
•27. 



document that the ineligibility dettrarfn^tion was reviewed flfot later than 
12 awnths following such .deunilnation? ' \ , 

Show that the client was refemtd to other 49encies arid facilities, a^ appro- 
priete? - ^ , \ t ^ vr 

sn<^ that the action uken and outcfcne were rtported^to other agencies'', \ 
as appropriate? 



s) scCTKw vrir, pase-m^ 



• le^utrsd for sen forsance with af^Utforjs and SuWlinej. 

V 




O © 

© © . 
© © 
© © 

© © 



© 
©' 



® 
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4Sg 



I 



I 

f 



for questions 23 threusn *1. fUti* Utt^r with: 
• (T) . YES- . 
CD • « 

(3) . HOT APaiCABLE (S/A) 



ELIG. 



IWRP 



SUPP. 
EVAL. 



SEaiCM Vll: TESWHATim OF CASES • STATUSES 30 »nd 28 



t£S: NO 



vERlC 



(X) 



(X) 



(X) 



(X) 



(X) 



(X) 



Case Closed Sutas 30 or SUtus 23 - Mot' Rehibll 1 Uted 
Dogf 5A# CCS4 Ttccrd . . . * 
» 29. contiin * certification of Jwligibility? 

Xf YESI dc€t th£ o€rtifi^zian of i^ligibility ... v 
•j. indicate the dau of certification? 

♦i. IndutJe the signature of the ippropHjte sti^te «9«ncy staff wrt>er 
(designated certifying agertt}? 

•c. Include tw reasons for .detennlnatlcn of Inillglbllltx? 
i. indicate t!»€ client's na«? 

• 29. -show that suluble eroloysient cannot be achieved? 

• 30. Show that es3)loy=«nt resulted without benefit derived frcn W services? 

• 31. docy«nt that one of the following situations necessitated closing the case 

. as rc: renabtlltited: 

•j. yit client decided not to follow through? 
•b. ccnuct was lost with the client? 

•c. there ws new infornatlon or coqplications? * ' 

» 32. doc.«nt that ^en t.iere ^^aj ntft infonjatlon or coooli cations, the client 
was referred to mother agency, as appropri*te? 

•-33. docurant that -men the client roved to another state, 'effort was aade to 
continue or amnge VRjervlcei? 

• lA contain the rationale for the Ineligibility determination as an 

. ' to. the 3rograffl2 (rbr case closed SUtus 30 fro« Sutas 10, answer .VA.) 

• 35 snsw*tnat the Ineligibility determination was »de only J^'^J"'^'®"!*"!!*; 

tl^n *1th the client, or ai appropriate, with the pannt, guardian, or other 
represinutlve? 



• 36, accu«nt that the client'^iir^fied in writing of the cToiurt. action taken? 

* yt. doc:*»eflt that the cTient ims infonwd in writing of client rights and 

l?^dTe$? (For case closed Sutus 30 fro« Stttus 10, answer N/A.) 

o^a. the ngnt to acWiniitrativt review and fair hearing? 

•b. the rignt to- sartlcipace in the anrtuii review of the,inel1g1!inity 
detemtnaticn? 



• ?eoul'^<J for confornancz *iy»'^lulattons and Sal dell nes. 



o 








CD • 




o 


0 




0 


© 




O 


0 


0 


® 


© 


0 


O 


0 


.© 


Q) 


0 




(D 


© 


© 


0. 


© 

» 


0 
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o 


^0 


© 


0 


©^ 
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0 


■. © 


0 


0 




0 




0 


0 


© 




0 
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ELIG. 



IKRP 



SlW. 
EVAt. 



S£q:o« Vtl: JeSMIWION OF CAStS - STATUStS 30 tnd 2S (contlmjed) 



ANSUER 
-YES NO N/A 



(X) 



ERIC 



rtccrd . 



33. 


docja«flt jny iction and dtclston inwflvlriig the cl^tnt's rt<;ue$t for in 
.*<ioj nutritive rtvlew of Js«ncy jctlon or fifr hearing? 


0 




•0 


39. 


doctc^nt !hat the. deteri? nation that the client was no longer eligible 
was Ptvlewed not later than 12 sanths following such determination? 


0 


0 


• ■© 


40. 


Shew that the client was rtferrtd to other. agencies and factlltles, as 
appropriate? 


0 


•0 . 


© 


41. 


show taat the action taken and ootcome were reported to other agencies, as 
appropriate? 




© 


0 



GO ON TO ScCTlOH YIIl, ?AGe 33. 
^equlrsa for confornance with Regulations and Guidelines. 



-2?-- 



For questions <2 through 51 > pleise wsWer with: 
© • YES 
(D • NO 
• 0 . HOT APPtlCABU (H/A) 



ELIG. 


IWRP 


SUPP. 
EVAL. 






fX) 






(X) . 






(X> 






(X) 


! 




(X) 






(X) 


















(X) 












Cx) 






(X) 






(X) 






(X) ■ 






&) 






(X) 






CX), 






fX) , 









StCriOH Vlt: TOMIKATIOH OF CASES - STATUS 26 



Case Closed Status 26 - gthafaHltated 
Oo4M ths oai4T€cord ... 

• 42. conuin a closure sUt«ent is an u»en(lnien,t tq the WRF? 

//YES, cic?« tA* «tst*i#ns inoluU' ^ 

•1. a descrlotlon of the basis upon whit»j the client wai deterained to 
be rehabilitated? 

•b. the type of occupation? " . 

♦c. the length of tlrae the client has been engaged in the occupation? 

^d. a descrtpticn of xny it is deternfned t^iit the client has "nade a 
'^Satisfactory aojus^nt? ' ^ • 

•ft. /any plans for the provision of post-esroloywefit services after a 
^ suitable objective nas been acnicved? 

• i3. Show that the client was provided wlth^essentlal YR services. IncludlJig: . 

•a. an evaluation of rehiollitatlon potential? 
•b. counseling and guidance? 

• a shew that the client was provfied with appropriate W services in accordance - 

with the Um I 

• 45. show that the client' has cocoWted the total J' '^'•"^^^"^iSSnM 

crlbed in the client's rertaoiliution progr*". including aaendwents? 

• iS Show that the client has been placed -In atnployment consistent with the esioloy^ 

aient objective for which services r^ve prepared the client? 

• 47. Show that the suiuole «fnoloy«rtt objective achieved 9y the client has been 

jaintained for a «1niaue of 50 days? 
- 43. <JocjR>ent. that the client was infomed in writing of the cTosure action .taken? 

• 49. $h<yw that the action aken and outcane were rtportea to other agencies, as 
, approoriate? 

•'50. shew that the client was referred to other agencies and. fadVltles. as appro- 
, priau? 

• SI. Shew tnat as oart of the process of being detenifted rehattUiuted, the 

client was infonued ... 

•a. of tllglbllity to receive necessary postal oyment services? 

•h. 6f ai< purpose for such post-enjloywent services? 

•c. of iny.pla'ns for sudr post-employwent servtcesJ 

M. tnat tne client snouJd contict the counselor. ^"j;';:^ f"* 

first /ear foi'.cwUg v:e (jeta-nmar.on is reiaoMitiud. before leav.. 
ifig» yie 400, or if prcolefls arise jeopamfi^.ng *j»e :oo. 

• ^equi rtd for canfomance wiVr Regulations imi Sa1«lines. - 

• 30- 'H ' 



YES W 



ANSWER 



© ® 



® 




® 


© 


® JD 


■® 




©■ 






© ' 




© 


© 


© 


® 


© 




© . 


© 


© 


© 




© 


© 


© 


© 


© 


© 


© 


© 


© 


© • 



© © 



© 
© 



ERIC 



fotr Questions 52 Uirou^ 58» please «r^w«nw1tt): 

0 ■ LESS THAiN:ADHQUA7c 
» CD • AOEQUATE ^ 
. ^ (D - NOT APPLICABLE (M/A) 



ELIG. 


IWRP 


SUPP. 
EVAL. : 




\ 


- (X) 






(X) 










! 






■fx) 






(XT 






(X) 


















(X) 












'■(X) 




V 


fX) 






fX) 






(X) 






' (x> ■ 






.(X) - 






(X) 






(X) 






(X), 



SEaiCN VU: TERMlKATICrt OF CASES - STATUS 26 



, ANSWER 
LESS AOEO-.- . 



ERLC 



• 52 M show twt «ter« 1$ \ reUllonship between the senficts rerxJered and the 



\show twt :her« 1$ 
vocational ogfX^? 

• 53. show that substantial services (those having a dtscemft>le latoact on the*" 

client's eflsployabUlty) were rendered to the client? 

, .5i.' snow that subsuntial pUceawnt services were or^vlded .to the client to- 
ward tne acquisition of a suita&le occupation. Including; 

a. an evaluation of tn'e client's job readiness? 

tt. guidance m the development and execution of tlje*^cHenf s plan j 
for joo seeding activities? • 

c. Instruction In r-a)c1ng job appUdittons?'' 

' d. instmction in client conduct during Interviews? ^ * 

e. eooloyer conuct? 

f. client registration -with the State esiployment service? 

g. job analysis and -nodiflcatlon? 

n. cpnsulutlon wly^ employers or supervisors, as required? 

1. efforts in the/area of setectlve placoaent if the client is severely 
disabled? 

• 55. shew Wat th* fol lowing canditioni of a suiUble placec»ent have been 3>et.. 

Insofar as possible: 

' - , the client and the enployer are each japhfiid? 

•D. the client is aaintaining ade<fu*te Interpersonal, relationships? 

•c. the client Is uainuining acceptable behavior in the Job envfroh- 
^ .Tjent? 

•d. the occupation is consistent w*1th the client's caoacUfej and 
abi titles, considerlrtg air client's choice? , 

•e. the client possesses accepuole skills to perforn or cantlnue'we. 
work satisfactorily? 

. _ the OTloytnent and wonting conditions wflU nqjt aggravate ate cli- 

efit's'disaoi Hty? 

•g. the client's e^lovrt^nc situation ^111 nor jeooacP*** nealtft 
' or safety of the cljeot op-^ers.? - ^ , 

t.te emoloymenc ts* regular am» reasonaoiy penvwent? 

•1. tre sMent r«sSes a *age co^rtn^urits witn that-paid others for , 
i:aitar '^rK jncer tugai requirements? 

qe^u*-ld hr confomance wtn Regulations and Guide? ines. 

. -31. " 
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© 
© 
© 
©' 
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©. 

©I 
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ELIG. 


IWRP 


suppV 

EV\L. 


^fr/TTTftV VTf« TVaufNiTTOl OF TA^r^ . ^TATIK 7(t (continue) 


LESS 


ANSWER 
ADEQ. 










(X) 


3gv V4ll d^M ths coin»€lor dacwt€mazian in ths aeM4 rtcord . . . • 
• 56. support the rtason and justtficition for closing the c«e, Including: 
^n. the eflo1o>i»ent status of the client? 


0. ® 


! 






4 


(X) 


*t). the basis on which eaiployDieftt was detenined to be suitable? 


0 


CD 










(X) 


57; show that when the client accepts a job which fails to meet one or uore 
- ' of the suitabjlity. criteria (specified in 55 a-i above), the counselor has: 

a. explained the unsuitable aspeQt(s) of the job? 


0 


0 


© ; 










b. indicated that the client has JMde the lob choice in light of these 
facts? 


0 


CD 


0 








(X) 


c. infonaed the client of the Hght to reapply for services if the 
job proves unsuitaole or the need arises for further servij^s? 

53. support the justification for closing the case when suitably criteria were 
noc "net, incl^idlng the consideration of altemacives? 


0 
0 


0 
0 


0 
0 






• 


(X) 










'T^^ SO ON TO SECTICN VIII, PAS£.33. 
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• Required for conformance with ilegulatlons and Guidelines. 
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SECTION 'VIII J TIMELINESS -ifesEARCH AND tOMMENTS 
. A, TIMELINESS 



INSTRUCTIONS: As indicators of timeliness in the various phases in the rehabilitation process, consider where 
applicable the information listed below pertaining to each of the status increments in columns A, B, and C, 
when responding to' questions 1-3 • 



Ticie lapses between referral/application of the client and the initial client interview, 
the request for information, the receipt of information on the client, additional diag- 
nostic services, extended evaluation (if applicable) , and any gaps between or during 
these events and the final eligibility decision and notification. 

Time lapses between eligibility determination and IWRP approval, amount of frequency of 
client contact during plan development* 

Time lapses between IWRP approval and initiation of services, duration of service 
receipt, contact with service providers schedules for counseling, authorizations for 
payment/purchase, client counselor contact, job availability, employer contacts, case 
record documentation and regular intervals and gaps m the f)rocess especially between 
status 22, placement, and closure (where applicable) , 







ANSWER COLUMNS 






\ 


A(00-08/10) 


B(10-12) 


C(12 


-26/28/30) 




a 


Yes 


No 


N/A 


Yes 


No 


N/A 


Yes 


No N/A 


1. Was the client served in a timely manner? 


















(1 


.e., witno'jc undesirable or unintended 


0 






0 


© 






© ■ 


tine lapses) . * . . . 


© 






0 


2. If 


no, was tne aelay sufficiently explained 


0 




0 






©.. 






m 


the case record? 


0 


0 


0 


0 


0 0* 


3. If 


explained, was the delay 'caiised by: (answer each) 




















, :jo response by client? 


d' 


© 


0 


0 


© 


© 


0 


© 0 


b) 


Client indecision? 


© 


© 


0 


0 


© 




©,. 


0*0 


c) 


Client unavailability? 


0 


© 


© 


0 


© 


0 


0 


0 0 


d) 


No contact by counselor? 


0' 


© 


0 


© 


© 


0 


0 


© 0 


e^) 


No action by counselor?^ 


0 


© 


0- 


0. 


© 


0 


0. 


0 0 


5> 


No counselor assigned? 


-0- 




-©r 






-©r 




g. 


Administrative delays? 


0' 


© 


0 


■0 


© 


© 


0 


© © 


h) 


Lack of resources? 


■0 


© 


© 


0 


© 


0 


0 


0 0 


1) 


Delay in receiving reports? 


0 


© 


© 


0 


© 


0 


0 


0 © 


3) 


Interagency delays? 


0 


© 


© 


0 


© 


0 


(D 


0 0 


k) 


lAck of placement opportunities? ■ , 


0 


© 


© 


0. 


0 


© 


0. 


0 0, 


. 1) 


Other? (specify ) 


0- 


© 


© 


0 


© 


© 


0 0© 



A. Status 00-08/10 
^•jferral 
(inj Eligibility 



Status 10-12 
(Eligibilxty-Plan 
Approval) 
. St3tas 12-26/28/30 
fplan Approval- 
Clos jre) 



'Uecc ttffi'si'cce be»ow :c provide eny cormients or feelings you may have about 
tn^ race re^c'C As er. exj-.:je, voyr evaluation of the documentation mi.y re.lect 
rc--".'»Anc« ^'-t- zryc Feoe^ai KeCuUtions and Guidelines; however, you rr^y feel that 
•'•e'cve-eri picture cr tr.a: a specific area within the rehcbilitation process 
(•:.e., tne natu-€Ncr.c/cr sccpe'of services, the counselor/client relationship, the 
c-'^c^'.cr's c^'^ivitv, tne clarity or orcanization of the, case. record , etc.) de- 
serves co'nnientr'Th>s's:>ac;e r.sy also be used to document specific p/oblems encoun- 
te^'ec "in 'jte j^evlev..' . , ' 
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